---
title: 'dagster cli'
sidebar_position: 1000
title_meta: 'dagster cli API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster cli Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
# Dagster CLI
:::warning Superseded feature
The `dagster` CLI has been superseded by the [`dg` CLI](https://docs.dagster.io/api/clis/dg-cli/dg-cli-reference). While it is still available, we recommend using `dg` commands for new projects and workflows. See the [dg CLI reference](https://docs.dagster.io/api/clis/dg-cli/dg-cli-reference) for details.
:::
## dagster asset
Commands for working with Dagster assets.
```shell
dagster asset [OPTIONS] COMMAND [ARGS]...
```
Commands:
list
List assets
materialize
Execute a run to materialize a selection of assets
Clears the asset partitions status cache, which is used by the webserver to load partition
>
pages more quickly. The cache will be rebuilt the next time the partition pages are loaded,
if caching is enabled.
Usage:
>
dagster asset wipe-cache –all
dagster asset wipe-cache \
dagster asset wipe-cache \
## dagster debug
Commands for helping debug Dagster issues by dumping or loading artifacts from specific runs.
This can be used to send a file to someone like the Dagster team who doesn’t have direct access
to your instance to allow them to view the events and details of a specific run.
Debug files can be viewed using dagster-webserver-debug cli.
Debug files can also be downloaded from the Dagster UI.
```shell
dagster debug [OPTIONS] COMMAND [ARGS]...
```
Commands:
export
Export the relevant artifacts for a job run from the current instance in to a file.
import
Import the relevant artifacts from debug files in to the current instance.
## dagster definitions validate
The dagster definitions validate command loads and validate your Dagster definitions using a Dagster instance.
This command indicates which code locations contain errors, and which ones can be successfully loaded.
Code locations containing errors are considered invalid, otherwise valid.
When running, this command sets the environment variable DAGSTER_IS_DEFS_VALIDATION_CLI=1.
This environment variable can be used to control the behavior of your code in validation mode.
This command returns an exit code 1 when errors are found, otherwise an exit code 0.
This command should be run in a Python environment where the dagster package is installed.
```shell
dagster definitions validate [OPTIONS]
```
Options:
-v, --verbose
Show verbose stack traces, including system frames in stack traces.
--load-with-grpc
Load the code locations using a gRPC server, instead of in-process.
--log-format \
Format of the logs for dagster services
Default: `'colored'`Options: colored | json | rich
--log-level \
Set the log level for dagster services.
Default: `'info'`Options: critical | error | warning | info | debug
--empty-workspace
Allow an empty workspace
-w, --workspace \
Path to workspace file. Argument can be provided multiple times.
-d, --working-directory \
Specify working directory to use when loading the repository or job
-f, --python-file \
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
-m, --module-name \
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
--package-name \
Specify Python package where repository or job function lives
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--grpc-port \
Port to use to connect to gRPC server
--grpc-socket \
Named socket to use to connect to gRPC server
--grpc-host \
Host to use to connect to gRPC server, defaults to localhost
--use-ssl
Use a secure channel when connecting to the gRPC server
Environment variables:
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-definitions-validate-d)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-definitions-validate-f)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-definitions-validate-m)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-definitions-validate-autoload-defs-module-name)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-definitions-validate-package-name)
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-definitions-validate-a)
## dagster dev
Start a local deployment of Dagster, including dagster-webserver running on localhost and the dagster-daemon running in the background
```shell
dagster dev [OPTIONS]
```
Options:
--code-server-log-level \
Set the log level for code servers spun up by dagster services.
Default: `'warning'`Options: critical | error | warning | info | debug
--log-level \
Set the log level for dagster services.
Default: `'info'`Options: critical | error | warning | info | debug
--log-format \
Format of the logs for dagster services
Default: `'colored'`Options: colored | json | rich
-p, --port, --dagit-port \
Port to use for the Dagster webserver.
-h, --host, --dagit-host \
Host to use for the Dagster webserver.
--live-data-poll-rate \
Rate at which the dagster UI polls for updated asset data (in milliseconds)
Default: `'2000'`
--use-legacy-code-server-behavior
Use the legacy behavior of the daemon and webserver each starting up their own code server
-v, --verbose
Show verbose stack traces for errors in the code server.
--db-statement-timeout \
The timeout in milliseconds to set on database statements sent to the DagsterInstance. Not respected in all configurations.
--db-pool-recycle \
The maximum age of a connection to use from the sqlalchemy pool without connection recycling.
--db-pool-max-overflow \
The maximum overflow size of the sqlalchemy pool. Set to -1 to disable.
--use-ssl
Use a secure channel when connecting to the gRPC server
--grpc-host \
Host to use to connect to gRPC server, defaults to localhost
--grpc-socket \
Named socket to use to connect to gRPC server
--grpc-port \
Port to use to connect to gRPC server
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--package-name \
Specify Python package where repository or job function lives
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
-m, --module-name \
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
-f, --python-file \
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
-d, --working-directory \
Specify working directory to use when loading the repository or job
-w, --workspace \
Path to workspace file. Argument can be provided multiple times.
--empty-workspace
Allow an empty workspace
Environment variables:
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-dev-a)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-dev-package-name)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-dev-autoload-defs-module-name)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-dev-m)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-dev-f)
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-dev-d)
## dagster instance
Commands for working with the current Dagster instance.
```shell
dagster instance [OPTIONS] COMMAND [ARGS]...
```
Commands:
concurrency
Commands for working with the instance-wide op concurrency.
info
List the information about the current instance.
migrate
Automatically migrate an out of date instance.
reindex
Rebuild index over historical runs for performance.
## dagster job
Commands for working with Dagster jobs.
```shell
dagster job [OPTIONS] COMMAND [ARGS]...
```
Commands:
backfill
Backfill a partitioned job.
This commands targets a job. The job can be specified in a number of ways:
1. dagster job backfill -j \<\> (works if .workspace.yaml exists)
2. dagster job backfill -j \<\> -w path/to/workspace.yaml
3. dagster job backfill -f /path/to/file.py -a define_some_job
4. dagster job backfill -m a_module.submodule -a define_some_job
5. dagster job backfill -f /path/to/file.py -a define_some_repo -j \<\>
6. dagster job backfill -m a_module.submodule -a define_some_repo -j \<\>
execute
Execute a job.
This commands targets a job. The job can be specified in a number of ways:
1. dagster job execute -f /path/to/file.py -a define_some_job
2. dagster job execute -m a_module.submodule -a define_some_job
3. dagster job execute -f /path/to/file.py -a define_some_repo -j \<\>
4. dagster job execute -m a_module.submodule -a define_some_repo -j \<\>
launch
Launch a job using the run launcher configured on the Dagster instance.
This commands targets a job. The job can be specified in a number of ways:
1. dagster job launch -j \<\> (works if .workspace.yaml exists)
2. dagster job launch -j \<\> -w path/to/workspace.yaml
3. dagster job launch -f /path/to/file.py -a define_some_job
4. dagster job launch -m a_module.submodule -a define_some_job
5. dagster job launch -f /path/to/file.py -a define_some_repo -j \<\>
6. dagster job launch -m a_module.submodule -a define_some_repo -j \<\>
list
List the jobs in a repository. Can only use ONE of –workspace/-w, –python-file/-f, –module-name/-m, –grpc-port, –grpc-socket.
print
Print a job.
This commands targets a job. The job can be specified in a number of ways:
1. dagster job print -j \<\> (works if .workspace.yaml exists)
2. dagster job print -j \<\> -w path/to/workspace.yaml
3. dagster job print -f /path/to/file.py -a define_some_job
4. dagster job print -m a_module.submodule -a define_some_job
5. dagster job print -f /path/to/file.py -a define_some_repo -j \<\>
6. dagster job print -m a_module.submodule -a define_some_repo -j \<\>
scaffold_config
Scaffold the config for a job.
This commands targets a job. The job can be specified in a number of ways:
1. dagster job scaffold_config -f /path/to/file.py -a define_some_job
2. dagster job scaffold_config -m a_module.submodule -a define_some_job
3. dagster job scaffold_config -f /path/to/file.py -a define_some_repo -j \<\>
4. dagster job scaffold_config -m a_module.submodule -a define_some_repo -j \<\>
## dagster run
Commands for working with Dagster job runs.
```shell
dagster run [OPTIONS] COMMAND [ARGS]...
```
Commands:
delete
Delete a run by id and its associated event logs. Warning: Cannot be undone
list
List the runs in the current Dagster instance.
migrate-repository
Migrate the run history for a job from a historic repository to its current repository.
wipe
Eliminate all run history and event logs. Warning: Cannot be undone.
## dagster schedule
Commands for working with Dagster schedules.
```shell
dagster schedule [OPTIONS] COMMAND [ARGS]...
```
Commands:
debug
Debug information about the scheduler.
list
List all schedules that correspond to a repository.
logs
Get logs for a schedule.
preview
Preview changes that will be performed by dagster schedule up.
restart
Restart a running schedule.
start
Start an existing schedule.
stop
Stop an existing schedule.
wipe
Delete the schedule history and turn off all schedules.
## dagster sensor
Commands for working with Dagster sensors.
```shell
dagster sensor [OPTIONS] COMMAND [ARGS]...
```
Commands:
cursor
Set the cursor value for an existing sensor.
list
List all sensors that correspond to a repository.
preview
Preview an existing sensor execution.
start
Start an existing sensor.
stop
Stop an existing sensor.
## dagster project
Commands for bootstrapping new Dagster projects and code locations.
```shell
dagster project [OPTIONS] COMMAND [ARGS]...
```
Commands:
from-example
Download one of the official Dagster examples to the current directory. This CLI enables you to quickly bootstrap your project with an officially maintained example.
list-examples
List the examples that available to bootstrap with.
scaffold
Create a folder structure with a single Dagster code location and other files such as pyproject.toml. This CLI enables you to quickly start building a new Dagster project with everything set up.
scaffold-code-location
(DEPRECATED; Use dagster project scaffold –excludes README.md instead) Create a folder structure with a single Dagster code location, in the current directory. This CLI helps you to scaffold a new Dagster code location within a folder structure that includes multiple Dagster code locations.
scaffold-repository
(DEPRECATED; Use dagster project scaffold –excludes README.md instead) Create a folder structure with a single Dagster repository, in the current directory. This CLI helps you to scaffold a new Dagster repository within a folder structure that includes multiple Dagster repositories
## dagster-graphql
Run a GraphQL query against the dagster interface to a specified repository or pipeline/job.
Can only use ONE of –workspace/-w, –python-file/-f, –module-name/-m, –grpc-port, –grpc-socket.
Examples:
1. dagster-graphql
2. dagster-graphql -w path/to/workspace.yaml
3. dagster-graphql -f path/to/file.py -a define_repo
4. dagster-graphql -m some_module -a define_repo
5. dagster-graphql -f path/to/file.py -a define_pipeline
6. dagster-graphql -m some_module -a define_pipeline
```shell
dagster-graphql [OPTIONS]
```
Options:
--version
Show the version and exit.
-t, --text \
GraphQL document to execute passed as a string
--file \
GraphQL document to execute passed as a file
-p, --predefined \
GraphQL document to execute, from a predefined set provided by dagster-graphql.
Options: launchPipelineExecution
-v, --variables \
A JSON encoded string containing the variables for GraphQL execution.
-r, --remote \
A URL for a remote instance running dagster-webserver to send the GraphQL request to.
-o, --output \
A file path to store the GraphQL response to. This flag is useful when making pipeline/job execution queries, since pipeline/job execution causes logs to print to stdout and stderr.
--ephemeral-instance
Use an ephemeral DagsterInstance instead of resolving via DAGSTER_HOME
--use-ssl
Use a secure channel when connecting to the gRPC server
--grpc-host \
Host to use to connect to gRPC server, defaults to localhost
--grpc-socket \
Named socket to use to connect to gRPC server
--grpc-port \
Port to use to connect to gRPC server
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--package-name \
Specify Python package where repository or job function lives
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
-m, --module-name \
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
-f, --python-file \
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
-d, --working-directory \
Specify working directory to use when loading the repository or job
-w, --workspace \
Path to workspace file. Argument can be provided multiple times.
--empty-workspace
Allow an empty workspace
Environment variables:
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-graphql-a)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-graphql-package-name)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-graphql-autoload-defs-module-name)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-graphql-m)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-graphql-f)
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-graphql-d)
## dagster-webserver
Run dagster-webserver. Loads a code location.
Can only use ONE of –workspace/-w, –python-file/-f, –module-name/-m, –grpc-port, –grpc-socket.
Examples:
1. dagster-webserver (works if ./workspace.yaml exists)
2. dagster-webserver -w path/to/workspace.yaml
3. dagster-webserver -f path/to/file.py
4. dagster-webserver -f path/to/file.py -d path/to/working_directory
5. dagster-webserver -m some_module
6. dagster-webserver -f path/to/file.py -a define_repo
7. dagster-webserver -m some_module -a define_repo
8. dagster-webserver -p 3333
Options can also provide arguments via environment variables prefixed with DAGSTER_WEBSERVER.
For example, DAGSTER_WEBSERVER_PORT=3333 dagster-webserver
```shell
dagster-webserver [OPTIONS]
```
Options:
-h, --host \
Host to run server on
Default: `'127.0.0.1'`
-p, --port \
Port to run server on - defaults to 3000
-l, --path-prefix \
The path prefix where server will be hosted (eg: /dagster-webserver)
Default: `''`
--db-statement-timeout \
The timeout in milliseconds to set on database statements sent to the DagsterInstance. Not respected in all configurations.
Default: `15000`
--db-pool-recycle \
The maximum age of a connection to use from the sqlalchemy pool without connection recycling. Set to -1 to disable. Not respected in all configurations.
Default: `3600`
--db-pool-max-overflow \
The maximum overflow size of the sqlalchemy pool. Set to -1 to disable.Not respected in all configurations.
Default: `20`
--read-only
Start server in read-only mode, where all mutations such as launching runs and turning schedules on/off are turned off.
--suppress-warnings
Filter all warnings when hosting server.
--uvicorn-log-level, --log-level \
Set the log level for the uvicorn web server.
Default: `'warning'`Options: critical | error | warning | info | debug | trace
--dagster-log-level \
Set the log level for dagster log events.
Default: `'info'`Options: critical | error | warning | info | debug
--log-format \
Format of the log output from the webserver
Default: `'colored'`Options: colored | json | rich
--code-server-log-level \
Set the log level for any code servers spun up by the webserver.
Default: `'info'`Options: critical | error | warning | info | debug
--live-data-poll-rate \
Rate at which the dagster UI polls for updated asset data (in milliseconds)
Default: `2000`
--version
Show the version and exit.
--use-ssl
Use a secure channel when connecting to the gRPC server
--grpc-host \
Host to use to connect to gRPC server, defaults to localhost
--grpc-socket \
Named socket to use to connect to gRPC server
--grpc-port \
Port to use to connect to gRPC server
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--package-name \
Specify Python package where repository or job function lives
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
-m, --module-name \
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
-f, --python-file \
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
-d, --working-directory \
Specify working directory to use when loading the repository or job
-w, --workspace \
Path to workspace file. Argument can be provided multiple times.
--empty-workspace
Allow an empty workspace
Environment variables:
DAGSTER_WEBSERVER_LOG_LEVEL
>
Provide a default for [`--dagster-log-level`](#cmdoption-dagster-webserver-dagster-log-level)
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-webserver-a)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-webserver-package-name)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-webserver-autoload-defs-module-name)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-webserver-m)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-webserver-f)
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-webserver-d)
## dagster-daemon run
Run any daemons configured on the DagsterInstance.
```shell
dagster-daemon run [OPTIONS]
```
Options:
--code-server-log-level \
Set the log level for any code servers spun up by the daemon.
Default: `'warning'`Options: critical | error | warning | info | debug
--log-level \
Set the log level for any code servers spun up by the daemon.
Default: `'info'`Options: critical | error | warning | info | debug
--log-format \
Format of the log output from the webserver
Default: `'colored'`Options: colored | json | rich
--use-ssl
Use a secure channel when connecting to the gRPC server
--grpc-host \
Host to use to connect to gRPC server, defaults to localhost
--grpc-socket \
Named socket to use to connect to gRPC server
--grpc-port \
Port to use to connect to gRPC server
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--package-name \
Specify Python package where repository or job function lives
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
-m, --module-name \
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
-f, --python-file \
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
-d, --working-directory \
Specify working directory to use when loading the repository or job
-w, --workspace \
Path to workspace file. Argument can be provided multiple times.
--empty-workspace
Allow an empty workspace
Environment variables:
DAGSTER_DAEMON_LOG_LEVEL
>
Provide a default for [`--log-level`](#cmdoption-dagster-daemon-run-log-level)
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-daemon-run-a)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-daemon-run-package-name)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-daemon-run-autoload-defs-module-name)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-daemon-run-m)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-daemon-run-f)
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-daemon-run-d)
## dagster-daemon wipe
Wipe all heartbeats from storage.
```shell
dagster-daemon wipe [OPTIONS]
```
## dagster api grpc
Serve the Dagster inter-process API over GRPC
```shell
dagster api grpc [OPTIONS]
```
Options:
-p, --port \
Port over which to serve. You must pass one and only one of –port/-p or –socket/-s.
-s, --socket \
Serve over a UDS socket. You must pass one and only one of –port/-p or –socket/-s.
-h, --host \
Hostname at which to serve. Default is localhost.
-n, --max-workers, --max_workers \
Maximum number of (threaded) workers to use in the GRPC server
--heartbeat
If set, the GRPC server will shut itself down when it fails to receive a heartbeat after a timeout configurable with –heartbeat-timeout.
--heartbeat-timeout \
Timeout after which to shutdown if –heartbeat is set and a heartbeat is not received
--lazy-load-user-code
Wait until the first LoadRepositories call to actually load the repositories, instead of waiting to load them when the server is launched. Useful for surfacing errors when the server is managed directly from the Dagster UI.
--use-python-environment-entry-point
If this flag is set, the server will signal to clients that they should launch dagster commands using \ -m dagster, instead of the default dagster entry point. This is useful when there are multiple Python environments running in the same machine, so a single dagster entry point is not enough to uniquely determine the environment.
--empty-working-directory
Indicates that the working directory should be empty and should not set to the current directory as a default
--fixed-server-id \
[INTERNAL] This option should generally not be used by users. Internal param used by dagster to spawn a gRPC server with the specified server id.
--log-level \
Level at which to log output from the code server process
Default: `'info'`Options: critical | error | warning | info | debug
--log-format \
Format of the log output from the code server process
Default: `'colored'`Options: colored | json | rich
--container-image \
Container image to use to run code from this server.
--container-context \
Serialized JSON with configuration for any containers created to run the code from this server.
--inject-env-vars-from-instance
Whether to load env vars from the instance and inject them into the environment.
--location-name \
Name of the code location this server corresponds to.
--instance-ref \
[INTERNAL] Serialized InstanceRef to use for accessing the instance
--enable-metrics
[INTERNAL] Retrieves current utilization metrics from GRPC server.
--defs-state-info \
[INTERNAL] Serialized DefsStateInfo to use for the server.
-a, --attribute \
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
--package-name \
Specify Python package where repository or job function lives
--autoload-defs-module-name \
A module to import and recursively search through for definitions.
-m, --module-name \
Specify module where dagster definitions reside as top-level symbols/variables and load the module as a code location in the current python environment.
-f, --python-file \
Specify python file where dagster definitions reside as top-level symbols/variables and load the file as a code location in the current python environment.
-d, --working-directory \
Specify working directory to use when loading the repository or job
Environment variables:
DAGSTER_GRPC_PORT
>
Provide a default for [`--port`](#cmdoption-dagster-api-grpc-p)
DAGSTER_GRPC_SOCKET
>
Provide a default for [`--socket`](#cmdoption-dagster-api-grpc-s)
DAGSTER_GRPC_HOST
>
Provide a default for [`--host`](#cmdoption-dagster-api-grpc-h)
DAGSTER_GRPC_MAX_WORKERS
>
Provide a default for [`--max-workers`](#cmdoption-dagster-api-grpc-n)
DAGSTER_LAZY_LOAD_USER_CODE
>
Provide a default for [`--lazy-load-user-code`](#cmdoption-dagster-api-grpc-lazy-load-user-code)
DAGSTER_USE_PYTHON_ENVIRONMENT_ENTRY_POINT
>
Provide a default for [`--use-python-environment-entry-point`](#cmdoption-dagster-api-grpc-use-python-environment-entry-point)
DAGSTER_EMPTY_WORKING_DIRECTORY
>
Provide a default for [`--empty-working-directory`](#cmdoption-dagster-api-grpc-empty-working-directory)
DAGSTER_CONTAINER_IMAGE
>
Provide a default for [`--container-image`](#cmdoption-dagster-api-grpc-container-image)
DAGSTER_CONTAINER_CONTEXT
>
Provide a default for [`--container-context`](#cmdoption-dagster-api-grpc-container-context)
DAGSTER_INJECT_ENV_VARS_FROM_INSTANCE
>
Provide a default for [`--inject-env-vars-from-instance`](#cmdoption-dagster-api-grpc-inject-env-vars-from-instance)
DAGSTER_LOCATION_NAME
>
Provide a default for [`--location-name`](#cmdoption-dagster-api-grpc-location-name)
DAGSTER_INSTANCE_REF
>
Provide a default for [`--instance-ref`](#cmdoption-dagster-api-grpc-instance-ref)
DAGSTER_ENABLE_SERVER_METRICS
>
Provide a default for [`--enable-metrics`](#cmdoption-dagster-api-grpc-enable-metrics)
DAGSTER_ATTRIBUTE
>
Provide a default for [`--attribute`](#cmdoption-dagster-api-grpc-a)
DAGSTER_PACKAGE_NAME
>
Provide a default for [`--package-name`](#cmdoption-dagster-api-grpc-package-name)
DAGSTER_autoload_defs_module_name
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dagster-api-grpc-autoload-defs-module-name)
DAGSTER_MODULE_NAME
>
Provide a default for [`--module-name`](#cmdoption-dagster-api-grpc-m)
DAGSTER_PYTHON_FILE
>
Provide a default for [`--python-file`](#cmdoption-dagster-api-grpc-f)
DAGSTER_WORKING_DIRECTORY
>
Provide a default for [`--working-directory`](#cmdoption-dagster-api-grpc-d)
---
---
title: 'create-dagster cli'
sidebar_position: 1000
title_meta: 'create-dagster cli API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'create-dagster cli Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
# create-dagster CLI
## Installation
See the [Installation](https://docs.dagster.io/getting-started/installation) guide.
## Commands
### create-dagster project
Scaffold a new Dagster project at PATH. The name of the project will be the final component of PATH.
This command can be run inside or outside of a workspace directory. If run inside a workspace,
the project will be added to the workspace’s list of project specs.
“.” may be passed as PATH to create the new project inside the existing working directory.
Created projects will have the following structure:
```default
├── src
│ └── PROJECT_NAME
│ ├── __init__.py
│ ├── definitions.py
│ ├── defs
│ │ └── __init__.py
│ └── components
│ └── __init__.py
├── tests
│ └── __init__.py
└── pyproject.toml
```
The src.PROJECT_NAME.defs directory holds Python objects that can be targeted by the
dg scaffold command or have dg-inspectable metadata. Custom component types in the project
live in src.PROJECT_NAME.components. These types can be created with dg scaffold component.
```shell
create-dagster project [OPTIONS] PATH
```
Examples:
```default
create-dagster project PROJECT_NAME
Scaffold a new project in new directory PROJECT_NAME. Automatically creates directory
and parent directories.
create-dagster project .
Scaffold a new project in the CWD. The project name is taken from the last component of the CWD.
```
Options:
--uv-sync, --no-uv-sync
Preemptively answer the “Run uv sync?” prompt presented after project initialization.
--use-editable-dagster
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the DAGSTER_GIT_REPO_DIR environment variable.
--verbose
Enable verbose output for debugging.
Arguments:
PATH
Required argument
### create-dagster workspace
Initialize a new Dagster workspace.
The scaffolded workspace folder has the following structure:
```default
├── projects
│ └── Dagster projects go here
├── deployments
│ └── local
│ ├── pyproject.toml
│ └── uv.lock
└── dg.toml
```
```shell
create-dagster workspace [OPTIONS] PATH
```
Examples:
```default
create-dagster workspace WORKSPACE_NAME
Scaffold a new workspace in new directory WORKSPACE_NAME. Automatically creates directory
and parent directories.
create-dagster workspace .
Scaffold a new workspace in the CWD. The workspace name is the last component of the CWD.
```
Options:
--uv-sync, --no-uv-sync
Preemptively answer the “Run uv sync?” prompt presented after project initialization.
--use-editable-dagster
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the DAGSTER_GIT_REPO_DIR environment variable.
--verbose
Enable verbose output for debugging.
Arguments:
PATH
Required argument
---
---
title: 'dg api reference'
sidebar_position: 1000
title_meta: 'dg api reference API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dg api reference Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
# dg api reference
## dg api
Make REST-like API calls to Dagster Plus.
```shell
dg api [OPTIONS] COMMAND [ARGS]...
```
### agent
Manage agents in Dagster Plus.
```shell
dg api agent [OPTIONS] COMMAND [ARGS]...
```
#### get
Get detailed information about a specific agent.
```shell
dg api agent get [OPTIONS] AGENT_ID
```
Example:
```default
$ dg api agent get c0b1ab17-1d2e-4f5b-9c8a-3e8d2c5f7a91
Label: analytics-prod-1
ID: c0b1ab17-1d2e-4f5b-9c8a-3e8d2c5f7a91
Status: RUNNING
Last Heartbeat: 2026-05-06 18:42:11 UTC
Metadata:
version: 1.12.0
dagster_cloud_version: 1.12.0
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
AGENT_ID
Required argument
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-agent-get-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-agent-get-api-token)
#### list
List all agents in the organization.
```shell
dg api agent list [OPTIONS]
```
Example:
```default
$ dg api agent list
LABEL ID STATUS LAST HEARTBEAT
analytics-prod-1 c0b1ab17-1d2e-4f5b-9c8a-3e8d2c5f7a91 RUNNING 2026-05-06 18:42:11 UTC
ingest-prod 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18 RUNNING 2026-05-06 18:42:08 UTC
Agent ad9c7f2e ad9c7f2e-3b15-4a87-9d61-5c8b3e2f1a04 STOPPED 2026-05-05 22:17:35 UTC
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-agent-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-agent-list-api-token)
### alert-policy
Manage alert policies in Dagster Plus.
```shell
dg api alert-policy [OPTIONS] COMMAND [ARGS]...
```
#### list
List alert policies for a deployment.
```shell
dg api alert-policy list [OPTIONS]
```
Example:
```default
$ dg api alert-policy list
alert_policies:
- name: failed_run_alert
description: Alert on failed runs
enabled: true
alert_targets:
- email_target:
email_addresses:
- oncall@example.com
event_types:
- JOB_FAILURE
- name: long_running_alert
description: Alert when runs exceed 1 hour
enabled: true
alert_targets:
- slack_target:
channel: '#data-alerts'
event_types:
- JOB_LONG_RUNNING
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-alert-policy-list-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-alert-policy-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-alert-policy-list-api-token)
#### sync
Sync alert policies from a YAML file.
```shell
dg api alert-policy sync [OPTIONS] FILE_PATH
```
Example:
```default
$ dg api alert-policy sync alert_policies.yaml
Synced alert policies: failed_run_alert, long_running_alert
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
FILE_PATH
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-alert-policy-sync-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-alert-policy-sync-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-alert-policy-sync-api-token)
### artifact
Upload and download artifacts in Dagster Plus.
```shell
dg api artifact [OPTIONS] COMMAND [ARGS]...
```
#### download
Download an artifact from Dagster Plus.
KEY is the artifact key (e.g. “my-model/latest”).
PATH is the local file path to save to.
```shell
dg api artifact download [OPTIONS] KEY PATH
```
Example:
```default
$ dg api artifact download my-model/latest ./model.pkl --deployment prod
Downloaded artifact 'my-model/latest' from deployment 'prod' to model.pkl.
```
Options:
--deployment \
Deployment name for deployment-scoped artifacts. If omitted, artifact is organization-scoped.
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
KEY
Required argument
PATH
Required argument
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-artifact-download-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-artifact-download-api-token)
#### upload
Upload an artifact to Dagster Plus.
KEY is the artifact key (e.g. “my-model/latest”).
PATH is the local file path to upload.
```shell
dg api artifact upload [OPTIONS] KEY PATH
```
Example:
```default
$ dg api artifact upload my-model/latest ./model.pkl --deployment prod
Uploaded artifact 'my-model/latest' to deployment 'prod'.
```
Options:
--deployment \
Deployment name for deployment-scoped artifacts. If omitted, artifact is organization-scoped.
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
KEY
Required argument
PATH
Required argument
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-artifact-upload-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-artifact-upload-api-token)
### asset
Manage assets in Dagster Plus.
```shell
dg api asset [OPTIONS] COMMAND [ARGS]...
```
#### get
Get specific asset details.
```shell
dg api asset get [OPTIONS] ASSET_KEY
```
Example:
```default
$ dg api asset get dim_customers
Asset Key: dim_customers
ID: 7e4c8b2a-1f3d-4e6b-9c5a-8d2f1e3b4c7d
Description: Customer dimension table
Group: marts
Kinds: dbt, snowflake
Deps: stg_customers
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ASSET_KEY
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-get-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-get-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-get-api-token)
#### get-evaluations
Get automation condition evaluation records for an asset.
Evaluations are only recorded when at least one subcondition has a different
value than the previous evaluation.
```shell
dg api asset get-evaluations [OPTIONS] ASSET_KEY
```
Example:
```default
$ dg api asset get-evaluations dim_customers --limit 2
EVAL ID TIMESTAMP NUM REQUESTED RUN IDS
4218 2026-05-06 18:00:00 UTC 1 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
4196 2026-05-05 18:00:00 UTC 1 2a1f7b3c-9d8e-4c5b-8a6d-3f1e2b9c4d7a
```
Options:
--limit \
Max evaluations to return (default: 50, max: 1000)
--cursor \
Cursor for pagination (evaluation ID)
--include-nodes
Include the condition evaluation node tree (verbose)
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ASSET_KEY
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-get-evaluations-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-get-evaluations-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-get-evaluations-api-token)
#### get-events
Get materialization and observation events for an asset.
```shell
dg api asset get-events [OPTIONS] ASSET_KEY
```
Example:
```default
$ dg api asset get-events dim_customers --limit 3
TIMESTAMP LEVEL TYPE RUN ID PARTITION
2026-05-06 18:00:12 UTC INFO ASSET_MATERIALIZATION 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
2026-05-05 18:00:08 UTC INFO ASSET_MATERIALIZATION 2a1f7b3c-9d8e-4c5b-8a6d-3f1e2b9c4d7a
2026-05-04 18:00:15 UTC INFO ASSET_MATERIALIZATION 8c4d2e7f-1a9b-4e3d-7c5b-9f2a1d8e3b6c
```
Options:
--event-type \
Filter by event type (default: both)
Options: ASSET_MATERIALIZATION | ASSET_OBSERVATION
--limit \
Max events to return (default: 50, max: 1000)
--before \
Events before this ISO timestamp (e.g. 2024-01-15T00:00:00)
--partition \
Filter by partition key (repeatable)
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ASSET_KEY
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-get-events-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-get-events-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-get-events-api-token)
#### get-health
Get health and runtime status for an asset.
```shell
dg api asset get-health [OPTIONS] ASSET_KEY
```
Example:
```default
$ dg api asset get-health dim_customers
Asset Key: dim_customers
Asset Health: HEALTHY
Materialization Status: MATERIALIZED
Freshness Status: PASSING
Asset Checks Status: PASSED
Last Materialized: 2026-05-06 18:00:12 UTC
Latest Materialization: 2026-05-06 18:00:12 UTC
Latest Run ID: 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Total Checks: 3
Failed Checks: 0
Warning Checks: 0
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ASSET_KEY
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-get-health-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-get-health-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-get-health-api-token)
#### get-partition-status
Get partition materialization stats for an asset.
```shell
dg api asset get-partition-status [OPTIONS] ASSET_KEY
```
Example:
```default
$ dg api asset get-partition-status daily_orders
Materialized: 364
Failed: 1
In Progress: 0
Total: 365
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ASSET_KEY
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-get-partition-status-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-get-partition-status-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-get-partition-status-api-token)
#### list
List assets with pagination.
```shell
dg api asset list [OPTIONS]
```
Example:
```default
$ dg api asset list
ASSET KEY GROUP DESCRIPTION KINDS
raw_customers ingestion Customer records from Snowflake snowflake
stg_customers staging Cleaned customer records dbt, snowflake
dim_customers marts Customer dimension table dbt, snowflake
daily_orders marts Daily order aggregates dbt, snowflake
```
Options:
--limit \
Number of assets to return (default: 50, max: 1000)
--cursor \
Cursor for pagination
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-list-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-list-api-token)
### asset-check
Manage asset checks in Dagster Plus.
```shell
dg api asset-check [OPTIONS] COMMAND [ARGS]...
```
#### get-executions
Get execution history for an asset check.
```shell
dg api asset-check get-executions [OPTIONS]
```
Example:
```default
$ dg api asset-check get-executions --asset-key dim_customers --check-name not_null --limit 3
STATUS RUN_ID TIMESTAMP PARTITION
SUCCEEDED 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a 2026-05-06 18:00:14 UTC
SUCCEEDED 2a1f7b3c-9d8e-4c5b-8a6d-3f1e2b9c4d7a 2026-05-05 18:00:09 UTC
FAILED 8c4d2e7f-1a9b-4e3d-7c5b-9f2a1d8e3b6c 2026-05-04 18:00:16 UTC
```
Options:
Maximum number of executions to return (default: 25)
--cursor \
Pagination cursor
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-check-get-executions-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-check-get-executions-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-check-get-executions-api-token)
#### list
List asset checks for an asset.
```shell
dg api asset-check list [OPTIONS]
```
Example:
```default
$ dg api asset-check list --asset-key dim_customers
NAME BLOCKING DESCRIPTION
not_null Yes Customer ID must not be null
unique_customer_id Yes Customer ID must be unique
valid_email No Email must match RFC 5322
```
Options:
Maximum number of asset checks to return (default: 10)
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-asset-check-list-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-asset-check-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-asset-check-list-api-token)
### code-location
Manage code locations in Dagster Plus.
```shell
dg api code-location [OPTIONS] COMMAND [ARGS]...
```
#### add
Add or update a code location in the deployment.
```shell
dg api code-location add [OPTIONS] LOCATION_NAME
```
Example:
```default
$ dg api code-location add ingestion --image dagster/ingestion:1.12.0 --module-name ingestion.definitions
Added or updated code location 'ingestion'.
```
Options:
--image \
Docker image for the code location
-m, --module-name \
Python module name
-p, --package-name \
Python package name
-f, --python-file \
Python file path
-w, --working-directory \
Working directory
--executable-path \
Python executable path
-a, --attribute \
Attribute to load definitions from
--commit-hash \
Git commit hash
--git-url \
Git repository URL
--location-file, --from \
YAML file with location configuration
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
LOCATION_NAME
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-code-location-add-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-code-location-add-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-code-location-add-api-token)
#### delete
Delete a code location from the deployment.
```shell
dg api code-location delete [OPTIONS] LOCATION_NAME
```
Example:
```default
$ dg api code-location delete ingestion
Deleted code location 'ingestion'.
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
LOCATION_NAME
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-code-location-delete-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-code-location-delete-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-code-location-delete-api-token)
#### get
Get detailed information about a specific code location.
```shell
dg api code-location get [OPTIONS] LOCATION_NAME
```
Example:
```default
$ dg api code-location get ingestion
Name: ingestion
Image: dagster/ingestion:1.12.0
Module: ingestion.definitions
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
LOCATION_NAME
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-code-location-get-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-code-location-get-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-code-location-get-api-token)
#### list
List code locations in the deployment.
```shell
dg api code-location list [OPTIONS]
```
Example:
```default
$ dg api code-location list
NAME IMAGE STATUS
ingestion dagster/ingestion:1.12.0 LOADED
analytics dagster/analytics:1.12.0 LOADED
marts dagster/marts:1.11.5 LOAD_ERROR
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-code-location-list-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-code-location-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-code-location-list-api-token)
### deployment
Manage deployments in Dagster Plus.
```shell
dg api deployment [OPTIONS] COMMAND [ARGS]...
```
#### delete
Delete a deployment by name.
```shell
dg api deployment delete [OPTIONS] NAME
```
Example:
```default
$ dg api deployment delete pr-123-feature-branch
Name: pr-123-feature-branch
ID: 42
Type: BRANCH
```
Options:
--allow-delete-full-deployment
Allow deleting a production (full) deployment
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
NAME
Required argument
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-deployment-delete-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-deployment-delete-api-token)
#### get
Show detailed information about a specific deployment.
```shell
dg api deployment get [OPTIONS] NAME
```
Example:
```default
$ dg api deployment get prod
Name: prod
ID: 1
Type: PRODUCTION
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
NAME
Required argument
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-deployment-get-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-deployment-get-api-token)
#### list
List deployments in the organization.
```shell
dg api deployment list [OPTIONS]
```
Example:
```default
$ dg api deployment list
NAME ID TYPE
prod 1 PRODUCTION
staging 2 PRODUCTION
```
Options:
--json
Output in JSON format for machine readability
--type \
Type of deployments to list (default: production)
Options: production | branch | all
--pr-status \
Filter branch deployments by pull request status (only applies with –type branch or all)
Options: OPEN | CLOSED | MERGED
--response-schema
Print the JSON Schema of the response model and exit.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-deployment-list-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-deployment-list-api-token)
##### get
Get settings for a deployment.
```shell
dg api deployment settings get [OPTIONS]
```
Example:
```default
$ dg api deployment settings get
run_queue:
max_concurrent_runs: 10
tag_concurrency_limits: []
run_retries:
max_retries: 3
sso_default_role: VIEWER
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-deployment-settings-get-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-deployment-settings-get-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-deployment-settings-get-api-token)
##### set
Set deployment settings from a YAML file.
```shell
dg api deployment settings set [OPTIONS] FILE_PATH
```
Example:
```default
$ dg api deployment settings set deployment-settings.yaml
run_queue:
max_concurrent_runs: 25
tag_concurrency_limits: []
run_retries:
max_retries: 3
sso_default_role: VIEWER
```
Options:
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
FILE_PATH
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-deployment-settings-set-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-deployment-settings-set-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-deployment-settings-set-api-token)
### issue
Manage issues in Dagster Plus.
```shell
dg api issue [OPTIONS] COMMAND [ARGS]...
```
#### add-link
Add a run or asset link to an issue.
```shell
dg api issue add-link [OPTIONS] ISSUE_ID
```
Example:
```default
$ dg api issue add-link 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18 --run-id 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Title: Snowflake load failed for daily_orders
Status: OPEN
Created By: oncall@example.com
Run IDs: 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Description: Snowflake permission denied while loading partition 2026-05-06
```
Options:
--run-id \
Run ID to link to the issue
--asset-key \
Asset key to link to the issue (slash-separated path, e.g. ‘my/asset’)
--json
Output in JSON format for machine readability
--response-schema
Print the JSON Schema of the response model and exit.
-d, --deployment \
Deployment to target.
-o, --organization \
Organization to target.
--api-token \
Dagster Cloud API token.
--view-graphql
Print GraphQL queries and responses to stderr for debugging.
Arguments:
ISSUE_ID
Required argument
Environment variables:
DAGSTER_CLOUD_DEPLOYMENT
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-add-link-d)
DAGSTER_CLOUD_ORGANIZATION
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-add-link-o)
DAGSTER_CLOUD_API_TOKEN
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-add-link-api-token)
#### create
Create a new issue.
```shell
dg api issue create [OPTIONS]
```
Example:
```default
$ dg api issue create --title "Snowflake load failed" --description "Permission denied loading daily_orders"
Title: Snowflake load failed
Status: OPEN
Created By: oncall@example.com
Description: Permission denied loading daily_orders
```
Options:
--title \<a href="#cmdoption-dg-api-issue-create-title" class="hash-link"></a></Link></dt>
<dd>
<strong>Required</strong> Title of the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-description'>--description \<description><a href="#cmdoption-dg-api-issue-create-description" class="hash-link"></a></Link></dt>
<dd>
<strong>Required</strong> Description of the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-status'>--status \<status><a href="#cmdoption-dg-api-issue-create-status" class="hash-link"></a></Link></dt>
<dd>
Status of the issue. Defaults to ‘OPEN’
Options: OPEN | CLOSED | TRIAGE | CANCELED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-json'>--json<a href="#cmdoption-dg-api-issue-create-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-response-schema'>--response-schema<a href="#cmdoption-dg-api-issue-create-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-issue-create-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-issue-create-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-issue-create-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-create-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-issue-create-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-create-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-create-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-create-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-issue-get">
#### get
Get an issue by ID.
```shell
dg api issue get [OPTIONS] ISSUE_ID
```
Example:
```default
$ dg api issue get 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18
Title: Snowflake load failed for daily_orders
Status: OPEN
Created By: oncall@example.com
Run IDs: 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Asset Keys: daily_orders
Description: Snowflake permission denied while loading partition 2026-05-06
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-json'>--json<a href="#cmdoption-dg-api-issue-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-issue-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-issue-get-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-issue-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-issue-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-issue-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-get-arg-ISSUE_ID'>ISSUE_ID<a href="#cmdoption-dg-api-issue-get-arg-ISSUE_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-get-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-issue-list">
#### list
List issues with pagination and optional filtering.
```shell
dg api issue list [OPTIONS]
```
Example:
```default
$ dg api issue list --limit 3 --status OPEN
STATUS TITLE ID CREATED BY
OPEN Snowflake load failed for daily_orders 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18 oncall@example.com
OPEN Stale freshness on stg_customers ad9c7f2e-3b15-4a87-9d61-5c8b3e2f1a04 alice@example.com
OPEN dbt test failure: unique_customer_id c0b1ab17-1d2e-4f5b-9c8a-3e8d2c5f7a91 bob@example.com
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-limit'>--limit \<limit><a href="#cmdoption-dg-api-issue-list-limit" class="hash-link"></a></Link></dt>
<dd>
Number of issues to return (default: 10)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-cursor'>--cursor \<cursor><a href="#cmdoption-dg-api-issue-list-cursor" class="hash-link"></a></Link></dt>
<dd>
Cursor for pagination
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-status'>--status \<statuses><a href="#cmdoption-dg-api-issue-list-status" class="hash-link"></a></Link></dt>
<dd>
Filter by issue status. Repeatable.
Options: OPEN | CLOSED | TRIAGE | CANCELED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-created-after'>--created-after \<created_after><a href="#cmdoption-dg-api-issue-list-created-after" class="hash-link"></a></Link></dt>
<dd>
Filter issues created after this date (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-created-before'>--created-before \<created_before><a href="#cmdoption-dg-api-issue-list-created-before" class="hash-link"></a></Link></dt>
<dd>
Filter issues created before this date (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-json'>--json<a href="#cmdoption-dg-api-issue-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-issue-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-issue-list-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-issue-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-issue-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-issue-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-list-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-list-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-issue-remove-link">
#### remove-link
Remove a run or asset link from an issue.
```shell
dg api issue remove-link [OPTIONS] ISSUE_ID
```
Example:
```default
$ dg api issue remove-link 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18 --run-id 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Title: Snowflake load failed for daily_orders
Status: OPEN
Created By: oncall@example.com
Description: Snowflake permission denied while loading partition 2026-05-06
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-run-id'>--run-id \<run_id><a href="#cmdoption-dg-api-issue-remove-link-run-id" class="hash-link"></a></Link></dt>
<dd>
Run ID to unlink from the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-asset-key'>--asset-key \<asset_key><a href="#cmdoption-dg-api-issue-remove-link-asset-key" class="hash-link"></a></Link></dt>
<dd>
Asset key to unlink from the issue (slash-separated path, e.g. ‘my/asset’)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-json'>--json<a href="#cmdoption-dg-api-issue-remove-link-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-response-schema'>--response-schema<a href="#cmdoption-dg-api-issue-remove-link-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-issue-remove-link-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-issue-remove-link-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-issue-remove-link-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-issue-remove-link-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-remove-link-arg-ISSUE_ID'>ISSUE_ID<a href="#cmdoption-dg-api-issue-remove-link-arg-ISSUE_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-remove-link-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-remove-link-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-remove-link-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-issue-update">
#### update
Update an existing issue.
```shell
dg api issue update [OPTIONS] ISSUE_ID
```
Example:
```default
$ dg api issue update 7e2c44b9-8f1a-4d6e-b0c3-2a5f9d4e6b18 --status RESOLVED
Title: Snowflake load failed for daily_orders
Status: RESOLVED
Created By: oncall@example.com
Description: Snowflake permission denied while loading partition 2026-05-06
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-status'>--status \<status><a href="#cmdoption-dg-api-issue-update-status" class="hash-link"></a></Link></dt>
<dd>
New status for the issue
Options: OPEN | CLOSED | TRIAGE | CANCELED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-title'>--title \<title><a href="#cmdoption-dg-api-issue-update-title" class="hash-link"></a></Link></dt>
<dd>
New title for the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-description'>--description \<description><a href="#cmdoption-dg-api-issue-update-description" class="hash-link"></a></Link></dt>
<dd>
New description for the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-context'>--context \<context><a href="#cmdoption-dg-api-issue-update-context" class="hash-link"></a></Link></dt>
<dd>
Additional context for the issue
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-json'>--json<a href="#cmdoption-dg-api-issue-update-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-response-schema'>--response-schema<a href="#cmdoption-dg-api-issue-update-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-issue-update-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-issue-update-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-issue-update-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-issue-update-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-issue-update-arg-ISSUE_ID'>ISSUE_ID<a href="#cmdoption-dg-api-issue-update-arg-ISSUE_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-issue-update-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-issue-update-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-issue-update-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-job">
### job
Manage jobs in Dagster Plus.
```shell
dg api job [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-job-get">
#### get
Get specific job details.
```shell
dg api job get [OPTIONS] JOB_NAME
```
Example:
```default
$ dg api job get ingest_customers
Name: ingest_customers
Description: Daily customer ingest
Asset Job: No
Tags: team=data-platform
Schedule: daily_customer_ingest (0 6 * * *) [RUNNING]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-json'>--json<a href="#cmdoption-dg-api-job-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-job-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-job-get-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-job-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-job-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-job-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-get-arg-JOB_NAME'>JOB_NAME<a href="#cmdoption-dg-api-job-get-arg-JOB_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-job-get-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-job-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-job-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-job-list">
#### list
List jobs in the deployment.
```shell
dg api job list [OPTIONS]
```
Example:
```default
$ dg api job list
NAME DESCRIPTION SCHEDULES SENSORS ASSET JOB
__ASSET_JOB 1 0 Yes
ingest_customers Daily customer ingest 1 0 No
retrain_recommendation_model Weekly model retrain 0 1 No
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-json'>--json<a href="#cmdoption-dg-api-job-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-job-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-job-list-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-job-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-job-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-job-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-job-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-job-list-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-job-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-job-list-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-organization">
### organization
Manage organization in Dagster Plus.
```shell
dg api organization [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-organization-saml">
#### saml
Manage SAML SSO configuration.
```shell
dg api organization saml [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-organization-saml-remove">
##### remove
Remove identity provider SAML metadata to disable SSO.
```shell
dg api organization saml remove [OPTIONS]
```
Example:
```default
$ dg api organization saml remove
The identity provider metadata was successfully removed.
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-remove-json'>--json<a href="#cmdoption-dg-api-organization-saml-remove-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-remove-response-schema'>--response-schema<a href="#cmdoption-dg-api-organization-saml-remove-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-remove-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-organization-saml-remove-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-remove-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-organization-saml-remove-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-remove-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-organization-saml-remove-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-organization-saml-remove-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-organization-saml-remove-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-organization-saml-upload">
##### upload
Upload identity provider SAML metadata to enable SSO.
```shell
dg api organization saml upload [OPTIONS] METADATA_FILE_PATH
```
Example:
```default
$ dg api organization saml upload idp-metadata.xml
The identity provider metadata was successfully uploaded.
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-json'>--json<a href="#cmdoption-dg-api-organization-saml-upload-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-response-schema'>--response-schema<a href="#cmdoption-dg-api-organization-saml-upload-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-organization-saml-upload-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-organization-saml-upload-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-organization-saml-upload-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-saml-upload-arg-METADATA_FILE_PATH'>METADATA_FILE_PATH<a href="#cmdoption-dg-api-organization-saml-upload-arg-METADATA_FILE_PATH" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-organization-saml-upload-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-organization-saml-upload-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-organization-settings">
#### settings
Manage organization settings.
```shell
dg api organization settings [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-organization-settings-get">
##### get
Get settings for the organization.
```shell
dg api organization settings get [OPTIONS]
```
Example:
```default
$ dg api organization settings get
sso_default_role: VIEWER
domain_allowlist:
- example.com
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-get-json'>--json<a href="#cmdoption-dg-api-organization-settings-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-organization-settings-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-organization-settings-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-organization-settings-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-organization-settings-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-organization-settings-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-organization-settings-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-organization-settings-set">
##### set
Set organization settings from a YAML file.
```shell
dg api organization settings set [OPTIONS] FILE_PATH
```
Example:
```default
$ dg api organization settings set organization-settings.yaml
sso_default_role: EDITOR
domain_allowlist:
- example.com
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-json'>--json<a href="#cmdoption-dg-api-organization-settings-set-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-response-schema'>--response-schema<a href="#cmdoption-dg-api-organization-settings-set-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-organization-settings-set-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-organization-settings-set-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-organization-settings-set-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-organization-settings-set-arg-FILE_PATH'>FILE_PATH<a href="#cmdoption-dg-api-organization-settings-set-arg-FILE_PATH" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-organization-settings-set-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-organization-settings-set-api-token)
</dd>
</dl>
</div></div></div>
<div class="section" id="dg-api-run">
### run
Manage runs in Dagster Plus.
```shell
dg api run [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-run-get">
#### get
Get run metadata by ID.
```shell
dg api run get [OPTIONS] RUN_ID
```
Example:
```default
$ dg api run get 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Run ID: 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Status: SUCCESS
Created: 2026-05-06 18:00:12 UTC
Started: 2026-05-06 18:00:14 UTC
Ended: 2026-05-06 18:04:42 UTC
Pipeline: ingest_customers
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-json'>--json<a href="#cmdoption-dg-api-run-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-run-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-run-get-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-run-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-run-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-run-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-arg-RUN_ID'>RUN_ID<a href="#cmdoption-dg-api-run-get-arg-RUN_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-run-get-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-run-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-run-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-run-get-events">
#### get-events
Get execution log events for a specific run ID.
```shell
dg api run get-events [OPTIONS] RUN_ID
```
Example:
```default
$ dg api run get-events 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a --limit 4
Logs for run 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a:
TIMESTAMP LEVEL STEP_KEY MESSAGE
--------------------------------------------------------------------------------
2026-05-06 18:00:14 INFO Started execution of run for "ingest_customers".
2026-05-06 18:00:14 INFO ingest_files Started execution of step "ingest_files".
2026-05-06 18:04:40 INFO ingest_files Finished execution of step "ingest_files" in 4m26s.
2026-05-06 18:04:42 INFO Finished execution of run for "ingest_customers".
Total log entries: 4
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-level'>--level \<levels><a href="#cmdoption-dg-api-run-get-events-level" class="hash-link"></a></Link></dt>
<dd>
Filter by log level (DEBUG, INFO, WARNING, ERROR, CRITICAL). Repeatable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-event-type'>--event-type \<event_types><a href="#cmdoption-dg-api-run-get-events-event-type" class="hash-link"></a></Link></dt>
<dd>
Filter by event type (e.g. STEP_FAILURE, RUN_START). Repeatable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-step'>--step \<step_keys><a href="#cmdoption-dg-api-run-get-events-step" class="hash-link"></a></Link></dt>
<dd>
Filter by step key (partial matching). Repeatable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-limit'>--limit \<limit><a href="#cmdoption-dg-api-run-get-events-limit" class="hash-link"></a></Link></dt>
<dd>
Maximum number of log entries to return
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-cursor'>--cursor \<after_cursor><a href="#cmdoption-dg-api-run-get-events-cursor" class="hash-link"></a></Link></dt>
<dd>
Pagination cursor for retrieving more logs
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-json'>--json<a href="#cmdoption-dg-api-run-get-events-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-response-schema'>--response-schema<a href="#cmdoption-dg-api-run-get-events-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-run-get-events-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-run-get-events-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-run-get-events-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-run-get-events-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-events-arg-RUN_ID'>RUN_ID<a href="#cmdoption-dg-api-run-get-events-arg-RUN_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-run-get-events-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-run-get-events-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-run-get-events-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-run-get-logs">
#### get-logs
Get stdout/stderr compute logs for a specific run.
```shell
dg api run get-logs [OPTIONS] RUN_ID
```
Example:
```default
$ dg api run get-logs 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a --step-key ingest_files
Compute logs for run 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a:
--- ingest_files [ingest_files] ---
STDOUT:
Loading 1,432 rows into raw_customers
Done.
STDERR:
(no errors)
Total steps with logs: 1
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-step-key'>--step-key \<step_key><a href="#cmdoption-dg-api-run-get-logs-step-key" class="hash-link"></a></Link></dt>
<dd>
Filter to a specific step
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-link-only'>--link-only<a href="#cmdoption-dg-api-run-get-logs-link-only" class="hash-link"></a></Link></dt>
<dd>
Return download URLs instead of log content
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-max-bytes'>--max-bytes \<max_bytes><a href="#cmdoption-dg-api-run-get-logs-max-bytes" class="hash-link"></a></Link></dt>
<dd>
Maximum bytes of log content per step
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-cursor'>--cursor \<cursor><a href="#cmdoption-dg-api-run-get-logs-cursor" class="hash-link"></a></Link></dt>
<dd>
Cursor for paginating log content
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-json'>--json<a href="#cmdoption-dg-api-run-get-logs-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-response-schema'>--response-schema<a href="#cmdoption-dg-api-run-get-logs-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-run-get-logs-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-run-get-logs-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-run-get-logs-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-run-get-logs-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-get-logs-arg-RUN_ID'>RUN_ID<a href="#cmdoption-dg-api-run-get-logs-arg-RUN_ID" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-run-get-logs-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-run-get-logs-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-run-get-logs-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-run-launch">
#### launch
Launch a run on a Dagster Plus deployment.
Use this to materialize assets or launch jobs against a deployed Dagster Plus
environment. For local in-process execution during development, use `dg launch`.
```shell
dg api run launch [OPTIONS]
```
Example:
```default
$ dg api run launch --location my_location --job ingest_customers
Run ID: 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
Status: STARTED
$ dg api run launch --location my_location \
--asset-key raw_customers --asset-key marts/dim_customers --wait
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-l'>-l, --location \<location><a href="#cmdoption-dg-api-run-launch-l" class="hash-link"></a></Link></dt>
<dd>
<strong>Required</strong> Code location name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-r'>-r, --repository \<repository><a href="#cmdoption-dg-api-run-launch-r" class="hash-link"></a></Link></dt>
<dd>
Repository name in the code location
Default: `'__repository__'`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-j'>-j, --job \<job_name><a href="#cmdoption-dg-api-run-launch-j" class="hash-link"></a></Link></dt>
<dd>
Name of the job to launch
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-asset-key'>--asset-key \<asset_keys><a href="#cmdoption-dg-api-run-launch-asset-key" class="hash-link"></a></Link></dt>
<dd>
Asset key to materialize. Use slash-separated syntax for prefixed keys (e.g. ‘my_prefix/my_asset’). Repeatable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-partition'>--partition \<partition><a href="#cmdoption-dg-api-run-launch-partition" class="hash-link"></a></Link></dt>
<dd>
Partition key. Partition ranges are not yet supported.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-tag'>--tag \<tag_options><a href="#cmdoption-dg-api-run-launch-tag" class="hash-link"></a></Link></dt>
<dd>
Tag to attach to the run as ‘key=value’. Repeatable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-config-json'>--config-json \<config_json><a href="#cmdoption-dg-api-run-launch-config-json" class="hash-link"></a></Link></dt>
<dd>
JSON string of run config to use for this run
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-w'>-w, --wait<a href="#cmdoption-dg-api-run-launch-w" class="hash-link"></a></Link></dt>
<dd>
Wait for the run to reach a terminal status before returning.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-i'>-i, --interval \<interval><a href="#cmdoption-dg-api-run-launch-i" class="hash-link"></a></Link></dt>
<dd>
Polling interval in seconds when –wait is set.
Default: `30`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-json'>--json<a href="#cmdoption-dg-api-run-launch-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-response-schema'>--response-schema<a href="#cmdoption-dg-api-run-launch-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-run-launch-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-run-launch-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-run-launch-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-launch-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-run-launch-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-run-launch-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-run-launch-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-run-launch-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-run-list">
#### list
List runs with optional filtering and pagination.
```shell
dg api run list [OPTIONS]
```
Example:
```default
$ dg api run list --limit 3
ID STATUS JOB CREATED
5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a SUCCESS ingest_customers 2026-05-06 18:00:12 UTC
2a1f7b3c-9d8e-4c5b-8a6d-3f1e2b9c4d7a SUCCESS ingest_customers 2026-05-05 18:00:08 UTC
8c4d2e7f-1a9b-4e3d-7c5b-9f2a1d8e3b6c FAILURE ingest_customers 2026-05-04 18:00:15 UTC
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-limit'>--limit \<limit><a href="#cmdoption-dg-api-run-list-limit" class="hash-link"></a></Link></dt>
<dd>
Number of runs to return (default: 50, max: 1000)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-cursor'>--cursor \<cursor><a href="#cmdoption-dg-api-run-list-cursor" class="hash-link"></a></Link></dt>
<dd>
Cursor for pagination (run ID)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-status'>--status \<statuses><a href="#cmdoption-dg-api-run-list-status" class="hash-link"></a></Link></dt>
<dd>
Filter by run status. Repeatable.
Options: QUEUED | NOT_STARTED | MANAGED | STARTING | STARTED | SUCCESS | FAILURE | CANCELING | CANCELED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-job'>--job \<job_name><a href="#cmdoption-dg-api-run-list-job" class="hash-link"></a></Link></dt>
<dd>
Filter by job name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-json'>--json<a href="#cmdoption-dg-api-run-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-run-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-run-list-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-run-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-run-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-run-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-run-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-run-list-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-run-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-run-list-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-schedule">
### schedule
Manage schedules in Dagster Plus.
```shell
dg api schedule [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-schedule-get">
#### get
Get specific schedule details.
```shell
dg api schedule get [OPTIONS] SCHEDULE_NAME
```
Example:
```default
$ dg api schedule get daily_customer_ingest
Name: daily_customer_ingest
Status: RUNNING
Cron Schedule: 0 6 * * *
Pipeline: ingest_customers
Description: Daily customer ingest at 06:00 UTC
Timezone: UTC
Next Tick: 2026-05-07 06:00:00 UTC
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-json'>--json<a href="#cmdoption-dg-api-schedule-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-schedule-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-schedule-get-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-schedule-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-schedule-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-schedule-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-arg-SCHEDULE_NAME'>SCHEDULE_NAME<a href="#cmdoption-dg-api-schedule-get-arg-SCHEDULE_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-schedule-get-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-schedule-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-schedule-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-schedule-get-ticks">
#### get-ticks
Get tick history for a specific schedule.
```shell
dg api schedule get-ticks [OPTIONS] SCHEDULE_NAME
```
Example:
```default
$ dg api schedule get-ticks daily_customer_ingest --limit 3
TIMESTAMP STATUS RUN IDS SKIP REASON
2026-05-06 06:00:00 UTC SUCCESS 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
2026-05-05 06:00:00 UTC SUCCESS 2a1f7b3c-9d8e-4c5b-8a6d-3f1e2b9c4d7a
2026-05-04 06:00:00 UTC FAILURE 8c4d2e7f-1a9b-4e3d-7c5b-9f2a1d8e3b6c
Total ticks: 3
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-status'>--status \<statuses><a href="#cmdoption-dg-api-schedule-get-ticks-status" class="hash-link"></a></Link></dt>
<dd>
Filter by tick status. Repeatable.
Options: STARTED | SKIPPED | SUCCESS | FAILURE
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-limit'>--limit \<limit><a href="#cmdoption-dg-api-schedule-get-ticks-limit" class="hash-link"></a></Link></dt>
<dd>
Maximum number of ticks to return
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-cursor'>--cursor \<cursor><a href="#cmdoption-dg-api-schedule-get-ticks-cursor" class="hash-link"></a></Link></dt>
<dd>
Pagination cursor
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-before'>--before \<before_timestamp><a href="#cmdoption-dg-api-schedule-get-ticks-before" class="hash-link"></a></Link></dt>
<dd>
Filter ticks before this Unix timestamp
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-after'>--after \<after_timestamp><a href="#cmdoption-dg-api-schedule-get-ticks-after" class="hash-link"></a></Link></dt>
<dd>
Filter ticks after this Unix timestamp
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-json'>--json<a href="#cmdoption-dg-api-schedule-get-ticks-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-response-schema'>--response-schema<a href="#cmdoption-dg-api-schedule-get-ticks-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-schedule-get-ticks-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-schedule-get-ticks-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-schedule-get-ticks-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-schedule-get-ticks-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-get-ticks-arg-SCHEDULE_NAME'>SCHEDULE_NAME<a href="#cmdoption-dg-api-schedule-get-ticks-arg-SCHEDULE_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-schedule-get-ticks-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-schedule-get-ticks-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-schedule-get-ticks-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-schedule-list">
#### list
List schedules in the deployment.
```shell
dg api schedule list [OPTIONS]
```
Example:
```default
$ dg api schedule list
NAME STATUS CRON PIPELINE
daily_customer_ingest RUNNING 0 6 * * * ingest_customers
hourly_event_aggregation RUNNING 0 * * * * aggregate_events
weekly_model_retrain STOPPED 0 9 * * MON retrain_recommendation_model
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-status'>--status \<status><a href="#cmdoption-dg-api-schedule-list-status" class="hash-link"></a></Link></dt>
<dd>
Filter schedules by status
Options: RUNNING | STOPPED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-json'>--json<a href="#cmdoption-dg-api-schedule-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-schedule-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-schedule-list-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-schedule-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-schedule-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-schedule-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-schedule-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-schedule-list-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-schedule-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-schedule-list-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-secret">
### secret
Manage secrets in Dagster Plus.
Secrets are environment variables that are encrypted and securely stored
in Dagster Plus. They can be scoped to different deployment levels and
code locations.
Security Note: Secret values are hidden by default. Use appropriate flags
and caution when displaying sensitive values.
```shell
dg api secret [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-secret-get">
#### get
Get details for a specific secret.
By default, the secret value is not shown for security reasons.
Use –show-value flag to display the actual secret value.
WARNING: When using –show-value, the secret will be visible in your terminal
and may be stored in shell history. Use with caution.
```shell
dg api secret get [OPTIONS] SECRET_NAME
```
Example:
```default
$ dg api secret get SNOWFLAKE_USER
Name: SNOWFLAKE_USER
Locations: All code locations
Scopes: Full Deployment, All Branch Deployments, Local Deployment
Permissions:
Can Edit: Yes
Can View Value: Yes
Value: <hidden - use --show-value to display>
Updated By: colton@dagsterlabs.com
Updated: 2023-05-04 19:14:15 UTC
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-location'>--location \<location><a href="#cmdoption-dg-api-secret-get-location" class="hash-link"></a></Link></dt>
<dd>
Filter by code location name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-show-value'>--show-value<a href="#cmdoption-dg-api-secret-get-show-value" class="hash-link"></a></Link></dt>
<dd>
Include secret value in output (use with caution - values will be visible in terminal)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-json'>--json<a href="#cmdoption-dg-api-secret-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-secret-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-secret-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-secret-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-secret-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-get-arg-SECRET_NAME'>SECRET_NAME<a href="#cmdoption-dg-api-secret-get-arg-SECRET_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-secret-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-secret-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-secret-list">
#### list
List secrets in the organization.
By default, secret values are not shown for security reasons.
Use ‘dg api secret get NAME –show-value’ to view specific values.
```shell
dg api secret list [OPTIONS]
```
Example:
```default
$ dg api secret list
NAME LOCATIONS SCOPES UPDATED
SNOWFLAKE_USER All Full Deployment, All Branch Deployments, Local Deployment 2023-05-04 19:14:15 UTC
SNOWFLAKE_PASSWORD All Full Deployment, All Branch Deployments, Local Deployment 2023-11-29 22:38:48 UTC
DATABRICKS_TOKEN All Full Deployment, Local Deployment 2025-07-25 23:52:00 UTC
GITHUB_TOKEN analytics Full Deployment 2025-08-11 18:19:12 UTC
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-location'>--location \<location><a href="#cmdoption-dg-api-secret-list-location" class="hash-link"></a></Link></dt>
<dd>
Filter secrets by code location name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-scope'>--scope \<scope><a href="#cmdoption-dg-api-secret-list-scope" class="hash-link"></a></Link></dt>
<dd>
Filter secrets by scope
Options: deployment | organization
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-json'>--json<a href="#cmdoption-dg-api-secret-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-secret-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-secret-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-secret-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-secret-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-secret-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-secret-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-secret-list-api-token)
</dd>
</dl>
</div></div>
<div class="section" id="dg-api-sensor">
### sensor
Manage sensors in Dagster Plus.
```shell
dg api sensor [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-api-sensor-get">
#### get
Get specific sensor details.
```shell
dg api sensor get [OPTIONS] SENSOR_NAME
```
Example:
```default
$ dg api sensor get new_file_sensor
Name: new_file_sensor
Status: RUNNING
Type: STANDARD
Description: Triggers ingest_customers when a new file lands in S3
Next Tick: 2026-05-06 18:43:00 UTC
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-json'>--json<a href="#cmdoption-dg-api-sensor-get-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-response-schema'>--response-schema<a href="#cmdoption-dg-api-sensor-get-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-sensor-get-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-sensor-get-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-sensor-get-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-sensor-get-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-arg-SENSOR_NAME'>SENSOR_NAME<a href="#cmdoption-dg-api-sensor-get-arg-SENSOR_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-sensor-get-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-sensor-get-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-sensor-get-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-sensor-get-ticks">
#### get-ticks
Get tick history for a specific sensor.
```shell
dg api sensor get-ticks [OPTIONS] SENSOR_NAME
```
Example:
```default
$ dg api sensor get-ticks new_file_sensor --limit 3
TIMESTAMP STATUS RUN IDS SKIP REASON
2026-05-06 18:42:00 UTC SUCCESS 5b3c8a91-2e4f-4d7b-9c6a-1f8d3e5b2c4a
2026-05-06 18:41:30 UTC SKIPPED - No new files in s3://incoming/customers/
2026-05-06 18:41:00 UTC SKIPPED - No new files in s3://incoming/customers/
Total ticks: 3
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-status'>--status \<statuses><a href="#cmdoption-dg-api-sensor-get-ticks-status" class="hash-link"></a></Link></dt>
<dd>
Filter by tick status. Repeatable.
Options: STARTED | SKIPPED | SUCCESS | FAILURE
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-limit'>--limit \<limit><a href="#cmdoption-dg-api-sensor-get-ticks-limit" class="hash-link"></a></Link></dt>
<dd>
Maximum number of ticks to return
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-cursor'>--cursor \<cursor><a href="#cmdoption-dg-api-sensor-get-ticks-cursor" class="hash-link"></a></Link></dt>
<dd>
Pagination cursor
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-before'>--before \<before_timestamp><a href="#cmdoption-dg-api-sensor-get-ticks-before" class="hash-link"></a></Link></dt>
<dd>
Filter ticks before this Unix timestamp
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-after'>--after \<after_timestamp><a href="#cmdoption-dg-api-sensor-get-ticks-after" class="hash-link"></a></Link></dt>
<dd>
Filter ticks after this Unix timestamp
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-json'>--json<a href="#cmdoption-dg-api-sensor-get-ticks-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-response-schema'>--response-schema<a href="#cmdoption-dg-api-sensor-get-ticks-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-sensor-get-ticks-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-sensor-get-ticks-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-sensor-get-ticks-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-sensor-get-ticks-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-get-ticks-arg-SENSOR_NAME'>SENSOR_NAME<a href="#cmdoption-dg-api-sensor-get-ticks-arg-SENSOR_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-sensor-get-ticks-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-sensor-get-ticks-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-sensor-get-ticks-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-api-sensor-list">
#### list
List sensors in the deployment.
```shell
dg api sensor list [OPTIONS]
```
Example:
```default
$ dg api sensor list
NAME STATUS TYPE
new_file_sensor RUNNING STANDARD
slack_alert_sensor RUNNING RUN_STATUS
retrain_trigger_sensor STOPPED ASSET
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-status'>--status \<status><a href="#cmdoption-dg-api-sensor-list-status" class="hash-link"></a></Link></dt>
<dd>
Filter sensors by status
Options: RUNNING | STOPPED
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-json'>--json<a href="#cmdoption-dg-api-sensor-list-json" class="hash-link"></a></Link></dt>
<dd>
Output in JSON format for machine readability
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-response-schema'>--response-schema<a href="#cmdoption-dg-api-sensor-list-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-api-sensor-list-d" class="hash-link"></a></Link></dt>
<dd>
Deployment to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-o'>-o, --organization \<organization><a href="#cmdoption-dg-api-sensor-list-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-api-token'>--api-token \<api_token><a href="#cmdoption-dg-api-sensor-list-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-api-sensor-list-view-graphql'>--view-graphql<a href="#cmdoption-dg-api-sensor-list-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-api-sensor-list-d)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-api-sensor-list-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-api-sensor-list-api-token)
</dd>
</dl>
</div></div></div></div>
---
---
title: 'dg cli local build command reference'
sidebar_position: 1000
title_meta: 'dg cli local build command reference API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dg cli local build command reference Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dg-cli-local-build-command-reference">
# dg CLI local build command reference
`dg` commands for scaffolding, checking, and listing Dagster entities, and running pipelines in a local Dagster instance.
<div class="section" id="dg-scaffold">
## dg scaffold
Commands for scaffolding Dagster entities.
```shell
dg scaffold [OPTIONS] COMMAND [ARGS]...
```
Commands:
<dl>
<dt>build-artifacts</dt>
<dd>
Scaffolds a Dockerfile to build the given Dagster project or workspace.
>
NOTE: This command is maintained for backward compatibility.
Consider using <cite>dg plus deploy configure [serverless|hybrid]</cite> instead for a complete
deployment setup including CI/CD configuration.
</dd>
</dl>
<dl>
<dt>component</dt>
<dd>
Scaffold of a custom Dagster component type.
>
This command must be run inside a Dagster project directory. The component type scaffold
will be placed in submodule <cite>\<project_name>.components.\<name></cite>.
</dd>
</dl>
<dl>
<dt>defs</dt>
<dd>
Commands for scaffolding Dagster code.
</dd>
</dl>
<dl>
<dt>github-actions</dt>
<dd>
Scaffold a GitHub Actions workflow for a Dagster project.
>
This command will create a GitHub Actions workflow in the <cite>.github/workflows</cite> directory.
NOTE: This command is maintained for backward compatibility.
Consider using <cite>dg plus deploy configure [serverless|hybrid] –git-provider github</cite>
instead for a complete deployment setup.
</dd>
</dl>
</div>
<div class="section" id="dg-dev">
## dg dev
Start a local instance of Dagster.
If run inside a workspace directory, this command will launch all projects in the
workspace. If launched inside a project directory, it will launch only that project.
```shell
dg dev [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-code-server-log-level'>--code-server-log-level \<code_server_log_level><a href="#cmdoption-dg-dev-code-server-log-level" class="hash-link"></a></Link></dt>
<dd>
Set the log level for code servers spun up by dagster services.
Default: `'warning'`Options: critical | error | warning | info | debug
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-log-level'>--log-level \<log_level><a href="#cmdoption-dg-dev-log-level" class="hash-link"></a></Link></dt>
<dd>
Set the log level for dagster services.
Default: `'info'`Options: critical | error | warning | info | debug
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-log-format'>--log-format \<log_format><a href="#cmdoption-dg-dev-log-format" class="hash-link"></a></Link></dt>
<dd>
Format of the logs for dagster services
Default: `'colored'`Options: colored | json | rich
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-p'>-p, --port \<port><a href="#cmdoption-dg-dev-p" class="hash-link"></a></Link></dt>
<dd>
Port to use for the Dagster webserver.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-h'>-h, --host \<host><a href="#cmdoption-dg-dev-h" class="hash-link"></a></Link></dt>
<dd>
Host to use for the Dagster webserver.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-live-data-poll-rate'>--live-data-poll-rate \<live_data_poll_rate><a href="#cmdoption-dg-dev-live-data-poll-rate" class="hash-link"></a></Link></dt>
<dd>
Rate at which the dagster UI polls for updated asset data (in milliseconds)
Default: `2000`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-db-statement-timeout'>--db-statement-timeout \<db_statement_timeout><a href="#cmdoption-dg-dev-db-statement-timeout" class="hash-link"></a></Link></dt>
<dd>
The timeout in milliseconds to set on database statements sent to the DagsterInstance. Not respected in all configurations.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-db-pool-recycle'>--db-pool-recycle \<db_pool_recycle><a href="#cmdoption-dg-dev-db-pool-recycle" class="hash-link"></a></Link></dt>
<dd>
The maximum age of a connection to use from the sqlalchemy pool without connection recycling.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-db-pool-max-overflow'>--db-pool-max-overflow \<db_pool_max_overflow><a href="#cmdoption-dg-dev-db-pool-max-overflow" class="hash-link"></a></Link></dt>
<dd>
The maximum overflow size of the sqlalchemy pool. Set to -1 to disable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-check-yaml'>--check-yaml, --no-check-yaml<a href="#cmdoption-dg-dev-check-yaml" class="hash-link"></a></Link></dt>
<dd>
Whether to schema-check defs.yaml files for the project before starting the dev server.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-target-path'>--target-path \<target_path><a href="#cmdoption-dg-dev-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-verbose'>--verbose<a href="#cmdoption-dg-dev-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-use-active-venv'>--use-active-venv<a href="#cmdoption-dg-dev-use-active-venv" class="hash-link"></a></Link></dt>
<dd>
Use the active virtual environment as defined by $VIRTUAL_ENV for all projects instead of attempting to resolve individual project virtual environments.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-autoload-defs-module-name'>--autoload-defs-module-name \<autoload_defs_module_name><a href="#cmdoption-dg-dev-autoload-defs-module-name" class="hash-link"></a></Link></dt>
<dd>
A module to import and recursively search through for definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-m'>-m, --module-name \<module_name><a href="#cmdoption-dg-dev-m" class="hash-link"></a></Link></dt>
<dd>
Specify module or modules (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each module as a code location in the current python environment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-f'>-f, --python-file \<python_file><a href="#cmdoption-dg-dev-f" class="hash-link"></a></Link></dt>
<dd>
Specify python file or files (flag can be used multiple times) where dagster definitions reside as top-level symbols/variables and load each file as a code location in the current python environment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-d'>-d, --working-directory \<working_directory><a href="#cmdoption-dg-dev-d" class="hash-link"></a></Link></dt>
<dd>
Specify working directory to use when loading the repository or job
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-w'>-w, --workspace \<workspace><a href="#cmdoption-dg-dev-w" class="hash-link"></a></Link></dt>
<dd>
Path to workspace file. Argument can be provided multiple times.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-dev-empty-workspace'>--empty-workspace<a href="#cmdoption-dg-dev-empty-workspace" class="hash-link"></a></Link></dt>
<dd>
Allow an empty workspace
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_autoload_defs_module_name</dt>
<dd>
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dg-dev-autoload-defs-module-name)
</dd>
</dl>
<dl>
<dt>DAGSTER_MODULE_NAME</dt>
<dd>
>
Provide a default for [`--module-name`](#cmdoption-dg-dev-m)
</dd>
</dl>
<dl>
<dt>DAGSTER_PYTHON_FILE</dt>
<dd>
>
Provide a default for [`--python-file`](#cmdoption-dg-dev-f)
</dd>
</dl>
<dl>
<dt>DAGSTER_WORKING_DIRECTORY</dt>
<dd>
>
Provide a default for [`--working-directory`](#cmdoption-dg-dev-d)
</dd>
</dl>
</div>
<div class="section" id="dg-check">
## dg check
Commands for checking the integrity of your Dagster code.
```shell
dg check [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-check-defs">
### defs
Loads and validates your Dagster definitions using a Dagster instance.
If run inside a deployment directory, this command will launch all code locations in the
deployment. If launched inside a code location directory, it will launch only that code
location.
When running, this command sets the environment variable <cite>DAGSTER_IS_DEFS_VALIDATION_CLI=1</cite>.
This environment variable can be used to control the behavior of your code in validation mode.
This command returns an exit code 1 when errors are found, otherwise an exit code 0.
```shell
dg check defs [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-log-level'>--log-level \<log_level><a href="#cmdoption-dg-check-defs-log-level" class="hash-link"></a></Link></dt>
<dd>
Set the log level for dagster services.
Default: `'warning'`Options: critical | error | warning | info | debug
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-log-format'>--log-format \<log_format><a href="#cmdoption-dg-check-defs-log-format" class="hash-link"></a></Link></dt>
<dd>
Format of the logs for dagster services
Default: `'colored'`Options: colored | json | rich
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-check-yaml'>--check-yaml, --no-check-yaml<a href="#cmdoption-dg-check-defs-check-yaml" class="hash-link"></a></Link></dt>
<dd>
Whether to schema-check defs.yaml files for the project before loading and checking all definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-target-path'>--target-path \<target_path><a href="#cmdoption-dg-check-defs-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-verbose'>--verbose<a href="#cmdoption-dg-check-defs-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-defs-use-active-venv'>--use-active-venv<a href="#cmdoption-dg-check-defs-use-active-venv" class="hash-link"></a></Link></dt>
<dd>
Use the active virtual environment as defined by $VIRTUAL_ENV for all projects instead of attempting to resolve individual project virtual environments.
</dd>
</dl>
</div>
<div class="section" id="dg-check-toml">
### toml
Check TOML configuration files (dg.toml, pyproject.toml) for validity.
```shell
dg check toml [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-toml-verbose'>--verbose<a href="#cmdoption-dg-check-toml-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-toml-target-path'>--target-path \<target_path><a href="#cmdoption-dg-check-toml-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div>
<div class="section" id="dg-check-yaml">
### yaml
Check defs.yaml files against their schemas, showing validation errors.
```shell
dg check yaml [OPTIONS] [PATHS]...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-yaml-watch'>--watch<a href="#cmdoption-dg-check-yaml-watch" class="hash-link"></a></Link></dt>
<dd>
Watch for changes to the component files and re-validate them.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-yaml-validate-requirements'>--validate-requirements, --no-validate-requirements<a href="#cmdoption-dg-check-yaml-validate-requirements" class="hash-link"></a></Link></dt>
<dd>
Validate environment variables in requirements for all components in the given module.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-yaml-verbose'>--verbose<a href="#cmdoption-dg-check-yaml-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-yaml-target-path'>--target-path \<target_path><a href="#cmdoption-dg-check-yaml-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-check-yaml-arg-PATHS'>PATHS<a href="#cmdoption-dg-check-yaml-arg-PATHS" class="hash-link"></a></Link></dt>
<dd>
Optional argument(s)
</dd>
</dl>
</div></div>
<div class="section" id="dg-list">
## dg list
Commands for listing Dagster entities.
```shell
dg list [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-list-component-tree">
### component-tree
```shell
dg list component-tree [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-component-tree-output-file'>--output-file \<output_file><a href="#cmdoption-dg-list-component-tree-output-file" class="hash-link"></a></Link></dt>
<dd>
Write to file instead of stdout. If not specified, will write to stdout.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-component-tree-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-component-tree-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-component-tree-verbose'>--verbose<a href="#cmdoption-dg-list-component-tree-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-list-components">
### components
List all available Dagster component types in the current Python environment.
```shell
dg list components [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-components-p'>-p, --package \<package><a href="#cmdoption-dg-list-components-p" class="hash-link"></a></Link></dt>
<dd>
Filter by package name.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-components-json'>--json<a href="#cmdoption-dg-list-components-json" class="hash-link"></a></Link></dt>
<dd>
Output as JSON instead of a table.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-components-response-schema'>--response-schema<a href="#cmdoption-dg-list-components-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-components-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-components-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-components-verbose'>--verbose<a href="#cmdoption-dg-list-components-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-list-defs">
### defs
List registered Dagster definitions in the current project environment.
```shell
dg list defs [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-json'>--json<a href="#cmdoption-dg-list-defs-json" class="hash-link"></a></Link></dt>
<dd>
Output as JSON instead of a table.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-response-schema'>--response-schema<a href="#cmdoption-dg-list-defs-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-p'>-p, --path \<path><a href="#cmdoption-dg-list-defs-p" class="hash-link"></a></Link></dt>
<dd>
Path to the definitions to list.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-a'>-a, --assets \<assets><a href="#cmdoption-dg-list-defs-a" class="hash-link"></a></Link></dt>
<dd>
Asset selection to list.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-c'>-c, --columns \<columns><a href="#cmdoption-dg-list-defs-c" class="hash-link"></a></Link></dt>
<dd>
Columns to display. Either a comma-separated list of column names, or multiple invocations of the flag. Available columns: key, group, deps, kinds, description, tags, cron, is_executable
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-verbose'>--verbose<a href="#cmdoption-dg-list-defs-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-defs-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-defs-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div>
<div class="section" id="dg-list-envs">
### envs
List environment variables from the .env file of the current project.
```shell
dg list envs [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-envs-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-envs-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-envs-verbose'>--verbose<a href="#cmdoption-dg-list-envs-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-list-projects">
### projects
List projects in the current workspace or emit the current project directory.
```shell
dg list projects [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-projects-verbose'>--verbose<a href="#cmdoption-dg-list-projects-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-projects-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-projects-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div>
<div class="section" id="dg-list-registry-modules">
### registry-modules
List dg plugins and their corresponding objects in the current Python environment.
```shell
dg list registry-modules [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-registry-modules-json'>--json<a href="#cmdoption-dg-list-registry-modules-json" class="hash-link"></a></Link></dt>
<dd>
Output as JSON instead of a table.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-registry-modules-response-schema'>--response-schema<a href="#cmdoption-dg-list-registry-modules-response-schema" class="hash-link"></a></Link></dt>
<dd>
Print the JSON Schema of the response model and exit.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-registry-modules-target-path'>--target-path \<target_path><a href="#cmdoption-dg-list-registry-modules-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-list-registry-modules-verbose'>--verbose<a href="#cmdoption-dg-list-registry-modules-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div></div>
<div class="section" id="dg-launch">
## dg launch
Launch a Dagster run.
```shell
dg launch [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-assets'>--assets \<assets><a href="#cmdoption-dg-launch-assets" class="hash-link"></a></Link></dt>
<dd>
Comma-separated Asset selection to target
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-job'>--job \<job><a href="#cmdoption-dg-launch-job" class="hash-link"></a></Link></dt>
<dd>
Job to target
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-partition'>--partition \<partition><a href="#cmdoption-dg-launch-partition" class="hash-link"></a></Link></dt>
<dd>
Asset partition to target
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-partition-range'>--partition-range \<partition_range><a href="#cmdoption-dg-launch-partition-range" class="hash-link"></a></Link></dt>
<dd>
Asset partition range to target i.e. \<start>…\<end>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-config-json'>--config-json \<config_json><a href="#cmdoption-dg-launch-config-json" class="hash-link"></a></Link></dt>
<dd>
JSON string of config to use for the launched run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-c'>-c, --config \<config><a href="#cmdoption-dg-launch-c" class="hash-link"></a></Link></dt>
<dd>
Specify one or more run config files. These can also be file patterns. If more than one run config file is captured then those files are merged. Files listed first take precedence.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-target-path'>--target-path \<target_path><a href="#cmdoption-dg-launch-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-verbose'>--verbose<a href="#cmdoption-dg-launch-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-a'>-a, --attribute \<attribute><a href="#cmdoption-dg-launch-a" class="hash-link"></a></Link></dt>
<dd>
Attribute that is either a 1) repository or job or 2) a function that returns a repository or job
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-package-name'>--package-name \<package_name><a href="#cmdoption-dg-launch-package-name" class="hash-link"></a></Link></dt>
<dd>
Specify Python package where repository or job function lives
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-autoload-defs-module-name'>--autoload-defs-module-name \<autoload_defs_module_name><a href="#cmdoption-dg-launch-autoload-defs-module-name" class="hash-link"></a></Link></dt>
<dd>
A module to import and recursively search through for definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-m'>-m, --module-name \<module_name><a href="#cmdoption-dg-launch-m" class="hash-link"></a></Link></dt>
<dd>
Specify module where dagster definitions reside as top-level symbols/variables and load the module as a code location in the current python environment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-f'>-f, --python-file \<python_file><a href="#cmdoption-dg-launch-f" class="hash-link"></a></Link></dt>
<dd>
Specify python file where dagster definitions reside as top-level symbols/variables and load the file as a code location in the current python environment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-launch-d'>-d, --working-directory \<working_directory><a href="#cmdoption-dg-launch-d" class="hash-link"></a></Link></dt>
<dd>
Specify working directory to use when loading the repository or job
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_ATTRIBUTE</dt>
<dd>
>
Provide a default for [`--attribute`](#cmdoption-dg-launch-a)
</dd>
</dl>
<dl>
<dt>DAGSTER_PACKAGE_NAME</dt>
<dd>
>
Provide a default for [`--package-name`](#cmdoption-dg-launch-package-name)
</dd>
</dl>
<dl>
<dt>DAGSTER_autoload_defs_module_name</dt>
<dd>
>
Provide a default for [`--autoload-defs-module-name`](#cmdoption-dg-launch-autoload-defs-module-name)
</dd>
</dl>
<dl>
<dt>DAGSTER_MODULE_NAME</dt>
<dd>
>
Provide a default for [`--module-name`](#cmdoption-dg-launch-m)
</dd>
</dl>
<dl>
<dt>DAGSTER_PYTHON_FILE</dt>
<dd>
>
Provide a default for [`--python-file`](#cmdoption-dg-launch-f)
</dd>
</dl>
<dl>
<dt>DAGSTER_WORKING_DIRECTORY</dt>
<dd>
>
Provide a default for [`--working-directory`](#cmdoption-dg-launch-d)
</dd>
</dl>
</div>
<div class="section" id="dg-scaffold-defs-example">
## dg scaffold defs example
<strong>Note:</strong> Before scaffolding definitions with `dg`, you must [create a project](https://docs.dagster.io/guides/build/projects/creating-projects) with the [create-dagster CLI](https://docs.dagster.io/api/clis/create-dagster) and activate its virtual environment.
You can use the `dg scaffold defs` command to scaffold a new asset underneath the `defs` folder. In this example, we scaffold an asset named `my_asset.py` and write it to the `defs/assets` directory:
```bash
dg scaffold defs dagster.asset assets/my_asset.py
Creating a component at /.../my-project/src/my_project/defs/assets/my_asset.py.
```
Once the asset has been scaffolded, we can see that a new file has been added to `defs/assets`, and view its contents:
```bash
tree
.
├── pyproject.toml
├── src
│ └── my_project
│ ├── __init__.py
│ └── defs
│ ├── __init__.py
│ └── assets
│ └── my_asset.py
├── tests
│ └── __init__.py
└── uv.lock
```
```python
cat src/my_project/defs/assets/my_asset.py
import dagster as dg
@dg.asset
def my_asset(context: dg.AssetExecutionContext) -> dg.MaterializeResult: ...
```
<strong>Note:</strong> You can run `dg scaffold defs` from within any directory in your project and the resulting files will always be created in the `<project-name>/src/<project_name>/defs/` folder.
In the above example, the scaffolded asset contains a basic commented-out definition. You can replace this definition with working code:
```python
import dagster as dg
@dg.asset(group_name="my_group")
def my_asset(context: dg.AssetExecutionContext) -> None:
"""Asset that greets you."""
context.log.info("hi!")
```
To confirm that the new asset now appears in the list of definitions, run <cite>dg list defs</cite>:
```bash
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ my_asset │ my_group │ │ │ Asset that greets you. │ │
│ │ └──────────┴──────────┴──────┴───────┴────────────────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────────┘
```
</div></div>
---
---
title: 'dg plus reference'
sidebar_position: 1000
title_meta: 'dg plus reference API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dg plus reference Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dg-plus-reference">
# dg plus reference
<div class="section" id="dg-plus">
## dg plus
Commands for interacting with Dagster Plus.
```shell
dg plus [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-config">
### config
Commands for viewing and managing Dagster Plus configuration.
```shell
dg plus config [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-config-set">
#### set
Set Dagster Plus CLI configuration values.
```shell
dg plus config set [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-set-api-token'>--api-token \<api_token><a href="#cmdoption-dg-plus-config-set-api-token" class="hash-link"></a></Link></dt>
<dd>
API token for authentication (or set DAGSTER_CLOUD_API_TOKEN).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-set-o'>-o, --organization \<organization><a href="#cmdoption-dg-plus-config-set-o" class="hash-link"></a></Link></dt>
<dd>
Organization name (or set DAGSTER_CLOUD_ORGANIZATION).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-set-d'>-d, --deployment \<deployment><a href="#cmdoption-dg-plus-config-set-d" class="hash-link"></a></Link></dt>
<dd>
Default deployment (or set DAGSTER_CLOUD_DEPLOYMENT).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-set-url'>--url \<url><a href="#cmdoption-dg-plus-config-set-url" class="hash-link"></a></Link></dt>
<dd>
Direct URL override for the Dagster Plus instance.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-set-region'>--region \<region><a href="#cmdoption-dg-plus-config-set-region" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud region. ‘eu’ sets URL to [https://eu.dagster.cloud](https://eu.dagster.cloud).
Options: us | eu
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-plus-config-set-api-token)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-plus-config-set-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-plus-config-set-d)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-config-view">
#### view
View the current Dagster Plus CLI configuration.
```shell
dg plus config view [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-config-view-s'>-s, --show-token<a href="#cmdoption-dg-plus-config-view-s" class="hash-link"></a></Link></dt>
<dd>
Show the full user token instead of a censored version.
</dd>
</dl>
</div></div>
<div class="section" id="dg-plus-create">
### create
Commands for creating configuration in Dagster Plus.
```shell
dg plus create [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-create-ci-api-token">
#### ci-api-token
Create a Dagster Plus API token for CI.
```shell
dg plus create ci-api-token [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-ci-api-token-description'>--description \<description><a href="#cmdoption-dg-plus-create-ci-api-token-description" class="hash-link"></a></Link></dt>
<dd>
Description for the token
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-ci-api-token-verbose'>--verbose<a href="#cmdoption-dg-plus-create-ci-api-token-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-create-env">
#### env
Create or update an environment variable in Dagster Plus.
```shell
dg plus create env [OPTIONS] ENV_NAME [ENV_VALUE]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-from-local-env'>--from-local-env<a href="#cmdoption-dg-plus-create-env-from-local-env" class="hash-link"></a></Link></dt>
<dd>
Pull the environment variable value from your shell environment or project .env file.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-scope'>--scope \<scope><a href="#cmdoption-dg-plus-create-env-scope" class="hash-link"></a></Link></dt>
<dd>
The deployment scope to set the environment variable in. Defaults to all scopes.
Options: full | branch | local
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-global'>--global<a href="#cmdoption-dg-plus-create-env-global" class="hash-link"></a></Link></dt>
<dd>
Whether to set the environment variable at the deployment level, for all locations.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-y'>-y, --yes<a href="#cmdoption-dg-plus-create-env-y" class="hash-link"></a></Link></dt>
<dd>
Do not confirm the creation of the environment variable, if it already exists.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-create-env-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-verbose'>--verbose<a href="#cmdoption-dg-plus-create-env-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-arg-ENV_NAME'>ENV_NAME<a href="#cmdoption-dg-plus-create-env-arg-ENV_NAME" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-create-env-arg-ENV_VALUE'>ENV_VALUE<a href="#cmdoption-dg-plus-create-env-arg-ENV_VALUE" class="hash-link"></a></Link></dt>
<dd>
Optional argument
</dd>
</dl>
</div></div>
<div class="section" id="dg-plus-deploy">
### deploy
Deploy a project or workspace to Dagster Plus. Handles all state management for the deploy
session, building and pushing a new code artifact for each project.
To run a full end-to-end deploy, run <cite>dg plus deploy</cite>. This will start a new session, build
and push the image for the project or workspace, and inform Dagster+ to deploy the newly built
code.
Each of the individual stages of the deploy is also available as its own subcommand for additional
customization.
```shell
dg plus deploy [OPTIONS] COMMAND [ARGS]...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-deployment'>--deployment \<deployment><a href="#cmdoption-dg-plus-deploy-deployment" class="hash-link"></a></Link></dt>
<dd>
Name of the Dagster+ deployment to which to deploy (or use as the base deployment if deploying to a branch deployment). If not set, defaults to the value set by <cite>dg plus login</cite>.
Default: `'deployment'`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-organization'>--organization \<organization><a href="#cmdoption-dg-plus-deploy-organization" class="hash-link"></a></Link></dt>
<dd>
Dagster+ organization to which to deploy. If not set, defaults to the value set by <cite>dg plus login</cite>.
Default: `'organization'`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-strategy'>--build-strategy \<build_strategy><a href="#cmdoption-dg-plus-deploy-build-strategy" class="hash-link"></a></Link></dt>
<dd>
Build strategy used to build code locations. ‘docker’ builds a Docker image (required for Hybrid agents). ‘python-executable’ builds PEX files (Serverless agents only).
Options: docker | python-executable
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-pex-build-method'>--pex-build-method \<pex_build_method><a href="#cmdoption-dg-plus-deploy-pex-build-method" class="hash-link"></a></Link></dt>
<dd>
Build method for PEX dependencies. ‘docker-fallback’ tries local first then Docker (default), ‘local’ uses only the current environment, ‘docker’ uses only a Docker builder. Only applies when –build-strategy=python-executable.
Options: local | docker | docker-fallback
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-python-version'>--python-version \<python_version><a href="#cmdoption-dg-plus-deploy-python-version" class="hash-link"></a></Link></dt>
<dd>
Python version used to deploy the project. If not set, defaults to the calling process’s Python minor version.
Options: 3.9 | 3.10 | 3.11 | 3.12 | 3.13 | 3.14
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-deployment-type'>--deployment-type \<deployment_type_str><a href="#cmdoption-dg-plus-deploy-deployment-type" class="hash-link"></a></Link></dt>
<dd>
Whether to deploy to a full deployment or a branch deployment. If unset, will attempt to infer from the current git branch.
Options: full | branch
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-agent-type'>--agent-type \<agent_type_str><a href="#cmdoption-dg-plus-deploy-agent-type" class="hash-link"></a></Link></dt>
<dd>
Whether this a Hybrid or serverless code location.
Options: serverless | hybrid
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-y'>-y, --yes<a href="#cmdoption-dg-plus-deploy-y" class="hash-link"></a></Link></dt>
<dd>
Skip confirmation prompts.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-git-url'>--git-url \<git_url><a href="#cmdoption-dg-plus-deploy-git-url" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-commit-hash'>--commit-hash \<commit_hash><a href="#cmdoption-dg-plus-deploy-commit-hash" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-location-name'>--location-name \<location_names><a href="#cmdoption-dg-plus-deploy-location-name" class="hash-link"></a></Link></dt>
<dd>
Name of the code location to set the build output for. Defaults to the current project’s code location, or every project’s code location when run in a workspace.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-status-url'>--status-url \<status_url><a href="#cmdoption-dg-plus-deploy-status-url" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-snapshot-base-condition'>--snapshot-base-condition \<snapshot_base_condition_str><a href="#cmdoption-dg-plus-deploy-snapshot-base-condition" class="hash-link"></a></Link></dt>
<dd>
Options: on-create | on-update
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-skip-validation'>--skip-validation<a href="#cmdoption-dg-plus-deploy-skip-validation" class="hash-link"></a></Link></dt>
<dd>
Skip configuration validation checks (not recommended).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-use-editable-dagster'>--use-editable-dagster<a href="#cmdoption-dg-plus-deploy-use-editable-dagster" class="hash-link"></a></Link></dt>
<dd>
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the <cite>DAGSTER_GIT_REPO_DIR</cite> environment variable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-plus-deploy-deployment)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-plus-deploy-organization)
</dd>
</dl>
<dl>
<dt>DAGSTER_BUILD_STRATEGY</dt>
<dd>
>
Provide a default for [`--build-strategy`](#cmdoption-dg-plus-deploy-build-strategy)
</dd>
</dl>
<dl>
<dt>DAGSTER_PEX_BUILD_METHOD</dt>
<dd>
>
Provide a default for [`--pex-build-method`](#cmdoption-dg-plus-deploy-pex-build-method)
</dd>
</dl>
<div class="section" id="dg-plus-deploy-build-and-push">
#### build-and-push
Builds a Docker image to be deployed, and pushes it to the registry
that was configured when the deploy session was started.
```shell
dg plus deploy build-and-push [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-agent-type'>--agent-type \<agent_type_str><a href="#cmdoption-dg-plus-deploy-build-and-push-agent-type" class="hash-link"></a></Link></dt>
<dd>
Whether this a Hybrid or serverless code location.
Options: serverless | hybrid
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-build-strategy'>--build-strategy \<build_strategy><a href="#cmdoption-dg-plus-deploy-build-and-push-build-strategy" class="hash-link"></a></Link></dt>
<dd>
Build strategy used to build code locations. ‘docker’ builds a Docker image (required for Hybrid agents). ‘python-executable’ builds PEX files (Serverless agents only).
Options: docker | python-executable
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-pex-build-method'>--pex-build-method \<pex_build_method><a href="#cmdoption-dg-plus-deploy-build-and-push-pex-build-method" class="hash-link"></a></Link></dt>
<dd>
Build method for PEX dependencies. ‘docker-fallback’ tries local first then Docker (default), ‘local’ uses only the current environment, ‘docker’ uses only a Docker builder. Only applies when –build-strategy=python-executable.
Options: local | docker | docker-fallback
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-python-version'>--python-version \<python_version><a href="#cmdoption-dg-plus-deploy-build-and-push-python-version" class="hash-link"></a></Link></dt>
<dd>
Python version used to deploy the project. If not set, defaults to the calling process’s Python minor version.
Options: 3.9 | 3.10 | 3.11 | 3.12 | 3.13 | 3.14
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-location-name'>--location-name \<location_names><a href="#cmdoption-dg-plus-deploy-build-and-push-location-name" class="hash-link"></a></Link></dt>
<dd>
Name of the code location to set the build output for. Defaults to the current project’s code location, or every project’s code location when run in a workspace.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-use-editable-dagster'>--use-editable-dagster<a href="#cmdoption-dg-plus-deploy-build-and-push-use-editable-dagster" class="hash-link"></a></Link></dt>
<dd>
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the <cite>DAGSTER_GIT_REPO_DIR</cite> environment variable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-build-and-push-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-build-and-push-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-build-and-push-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_BUILD_STRATEGY</dt>
<dd>
>
Provide a default for [`--build-strategy`](#cmdoption-dg-plus-deploy-build-and-push-build-strategy)
</dd>
</dl>
<dl>
<dt>DAGSTER_PEX_BUILD_METHOD</dt>
<dd>
>
Provide a default for [`--pex-build-method`](#cmdoption-dg-plus-deploy-build-and-push-pex-build-method)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-configure">
#### configure
Scaffold deployment configuration files for Dagster Plus.
If no subcommand is specified, will attempt to auto-detect the agent type from your
Dagster Plus deployment. If detection fails, you will be prompted to choose between
serverless or hybrid.
If run in a [workspace](https://docs.dagster.io/guides/build/projects/workspaces/creating-workspaces),
will scaffold configuration files for the workspace and all projects contained in it.
```shell
dg plus deploy configure [OPTIONS] COMMAND [ARGS]...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-git-provider'>--git-provider \<git_provider><a href="#cmdoption-dg-plus-deploy-configure-git-provider" class="hash-link"></a></Link></dt>
<dd>
Git provider for CI/CD scaffolding
Options: github | gitlab
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-configure-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<div class="section" id="dg-plus-deploy-configure-hybrid">
##### hybrid
Scaffold deployment configuration for Dagster Plus Hybrid.
This creates:
- Dockerfile and build.yaml for containerization
- container_context.yaml with platform-specific config (k8s/ecs/docker)
- Required files for CI/CD based on your Git provider (GitHub Actions or GitLab CI)
```shell
dg plus deploy configure hybrid [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-git-provider'>--git-provider \<git_provider><a href="#cmdoption-dg-plus-deploy-configure-hybrid-git-provider" class="hash-link"></a></Link></dt>
<dd>
Git provider for CI/CD scaffolding
Options: github | gitlab
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-agent-platform'>--agent-platform \<agent_platform><a href="#cmdoption-dg-plus-deploy-configure-hybrid-agent-platform" class="hash-link"></a></Link></dt>
<dd>
Agent platform (k8s, ecs, or docker)
Options: k8s | ecs | docker
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-registry-url'>--registry-url \<registry_url><a href="#cmdoption-dg-plus-deploy-configure-hybrid-registry-url" class="hash-link"></a></Link></dt>
<dd>
Container registry URL for Docker images (e.g., 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-python-version'>--python-version \<python_version><a href="#cmdoption-dg-plus-deploy-configure-hybrid-python-version" class="hash-link"></a></Link></dt>
<dd>
Python version used to deploy the project
Options: 3.9 | 3.10 | 3.11 | 3.12 | 3.13 | 3.14
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-organization'>--organization \<organization><a href="#cmdoption-dg-plus-deploy-configure-hybrid-organization" class="hash-link"></a></Link></dt>
<dd>
Dagster Plus organization name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-url'>--url \<url><a href="#cmdoption-dg-plus-deploy-configure-hybrid-url" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud URL (defaults to [https://dagster.cloud](https://dagster.cloud))
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-deployment'>--deployment \<deployment><a href="#cmdoption-dg-plus-deploy-configure-hybrid-deployment" class="hash-link"></a></Link></dt>
<dd>
Deployment name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-git-root'>--git-root \<git_root><a href="#cmdoption-dg-plus-deploy-configure-hybrid-git-root" class="hash-link"></a></Link></dt>
<dd>
Path to the git repository root
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-y'>-y, --yes<a href="#cmdoption-dg-plus-deploy-configure-hybrid-y" class="hash-link"></a></Link></dt>
<dd>
Skip confirmation prompts
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-use-editable-dagster'>--use-editable-dagster<a href="#cmdoption-dg-plus-deploy-configure-hybrid-use-editable-dagster" class="hash-link"></a></Link></dt>
<dd>
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the <cite>DAGSTER_GIT_REPO_DIR</cite> environment variable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-hybrid-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-configure-hybrid-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-configure-serverless">
##### serverless
Scaffold deployment configuration for Dagster Plus Serverless.
This creates:
- Required files for CI/CD based on your Git provider (GitHub Actions or GitLab CI)
- Dockerfile and build.yaml for containerization (if –no-pex-deploy is used)
```shell
dg plus deploy configure serverless [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-git-provider'>--git-provider \<git_provider><a href="#cmdoption-dg-plus-deploy-configure-serverless-git-provider" class="hash-link"></a></Link></dt>
<dd>
Git provider for CI/CD scaffolding
Options: github | gitlab
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-python-version'>--python-version \<python_version><a href="#cmdoption-dg-plus-deploy-configure-serverless-python-version" class="hash-link"></a></Link></dt>
<dd>
Python version used to deploy the project
Options: 3.9 | 3.10 | 3.11 | 3.12 | 3.13 | 3.14
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-organization'>--organization \<organization><a href="#cmdoption-dg-plus-deploy-configure-serverless-organization" class="hash-link"></a></Link></dt>
<dd>
Dagster Plus organization name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-url'>--url \<url><a href="#cmdoption-dg-plus-deploy-configure-serverless-url" class="hash-link"></a></Link></dt>
<dd>
Dagster Plus URL for the organization
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-deployment'>--deployment \<deployment><a href="#cmdoption-dg-plus-deploy-configure-serverless-deployment" class="hash-link"></a></Link></dt>
<dd>
Deployment name
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-git-root'>--git-root \<git_root><a href="#cmdoption-dg-plus-deploy-configure-serverless-git-root" class="hash-link"></a></Link></dt>
<dd>
Path to the git repository root
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-pex-deploy'>--pex-deploy, --no-pex-deploy<a href="#cmdoption-dg-plus-deploy-configure-serverless-pex-deploy" class="hash-link"></a></Link></dt>
<dd>
Enable PEX-based fast deploys (default: True). If disabled, Docker builds will be used.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-y'>-y, --yes<a href="#cmdoption-dg-plus-deploy-configure-serverless-y" class="hash-link"></a></Link></dt>
<dd>
Skip confirmation prompts
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-use-editable-dagster'>--use-editable-dagster<a href="#cmdoption-dg-plus-deploy-configure-serverless-use-editable-dagster" class="hash-link"></a></Link></dt>
<dd>
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the <cite>DAGSTER_GIT_REPO_DIR</cite> environment variable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-configure-serverless-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-configure-serverless-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div></div>
<div class="section" id="dg-plus-deploy-deselect">
#### deselect
Mark specified locations as excluded from the current build session.
```shell
dg plus deploy deselect [OPTIONS] LOCATION_NAMES...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-deselect-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-deselect-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-deselect-arg-LOCATION_NAMES'>LOCATION_NAMES<a href="#cmdoption-dg-plus-deploy-deselect-arg-LOCATION_NAMES" class="hash-link"></a></Link></dt>
<dd>
Required argument(s)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-finish">
#### finish
Once all needed images have been built and pushed, completes the deploy session, signaling
to the Dagster+ API that the deployment can be updated to the newly built and pushed code.
```shell
dg plus deploy finish [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-finish-location-name'>--location-name \<location_names><a href="#cmdoption-dg-plus-deploy-finish-location-name" class="hash-link"></a></Link></dt>
<dd>
Name of the code location to set the build output for. Defaults to the current project’s code location, or every project’s code location when run in a workspace.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-finish-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-finish-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-finish-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-finish-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-inspect">
#### inspect
Print JSON info about the current CI/CD environment.
```shell
dg plus deploy inspect [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-inspect-project-dir'>--project-dir \<project_dir><a href="#cmdoption-dg-plus-deploy-inspect-project-dir" class="hash-link"></a></Link></dt>
<dd>
Project directory for CI context detection.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-inspect-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-inspect-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-list-locations">
#### list-locations
List all locations in the current deploy session with their selection status.
```shell
dg plus deploy list-locations [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-list-locations-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-list-locations-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-notify">
#### notify
Update a GitHub PR comment with deploy status.
```shell
dg plus deploy notify [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-notify-project-dir'>--project-dir \<project_dir><a href="#cmdoption-dg-plus-deploy-notify-project-dir" class="hash-link"></a></Link></dt>
<dd>
Project directory for CI context detection.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-notify-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-notify-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-refresh-defs-state">
#### refresh-defs-state
[Experimental] If using StateBackedComponents, this command will execute the <cite>refresh_state</cite> on each of them,
and set the defs_state_info for each location.
Environment variables from Dagster Plus are automatically fetched and injected into the
subprocess environment during state refresh. Uses fullDeploymentScope for production deploys
and allBranchDeploymentsScope for branch deploys.
```shell
dg plus deploy refresh-defs-state [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-refresh-defs-state-use-editable-dagster'>--use-editable-dagster<a href="#cmdoption-dg-plus-deploy-refresh-defs-state-use-editable-dagster" class="hash-link"></a></Link></dt>
<dd>
Install all Dagster package dependencies from a local Dagster clone. The location of the local Dagster clone will be read from the <cite>DAGSTER_GIT_REPO_DIR</cite> environment variable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-refresh-defs-state-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-refresh-defs-state-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-refresh-defs-state-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-refresh-defs-state-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-refresh-defs-state-management-type'>--management-type \<management_type><a href="#cmdoption-dg-plus-deploy-refresh-defs-state-management-type" class="hash-link"></a></Link></dt>
<dd>
Only refresh components with the specified management type. Can be specified multiple times to include multiple types. By default, refreshes VERSIONED_STATE_STORAGE and LOCAL_FILESYSTEM components.
Options: LOCAL_FILESYSTEM | VERSIONED_STATE_STORAGE
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-select">
#### select
Mark specified locations as included in the current build session.
```shell
dg plus deploy select [OPTIONS] LOCATION_NAMES...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-select-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-select-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-select-arg-LOCATION_NAMES'>LOCATION_NAMES<a href="#cmdoption-dg-plus-deploy-select-arg-LOCATION_NAMES" class="hash-link"></a></Link></dt>
<dd>
Required argument(s)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-set-build-output">
#### set-build-output
If building a Docker image was built outside of the <cite>dg</cite> CLI, configures the deploy session
to indicate the correct tag to use when the session is finished.
```shell
dg plus deploy set-build-output [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-set-build-output-image-tag'>--image-tag \<image_tag><a href="#cmdoption-dg-plus-deploy-set-build-output-image-tag" class="hash-link"></a></Link></dt>
<dd>
<strong>Required</strong> Tag for the built docker image.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-set-build-output-location-name'>--location-name \<location_names><a href="#cmdoption-dg-plus-deploy-set-build-output-location-name" class="hash-link"></a></Link></dt>
<dd>
Name of the code location to set the build output for. Defaults to the current project’s code location, or every project’s code location when run in a workspace.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-set-build-output-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-set-build-output-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-set-build-output-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-set-build-output-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-start">
#### start
Start a new deploy session. Determines which code locations will be deployed and what
deployment is being targeted (creating a new branch deployment if needed), and initializes a
folder on the filesystem where state about the deploy session will be stored.
```shell
dg plus deploy start [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-deployment'>--deployment \<deployment><a href="#cmdoption-dg-plus-deploy-start-deployment" class="hash-link"></a></Link></dt>
<dd>
Name of the Dagster+ deployment to which to deploy (or use as the base deployment if deploying to a branch deployment). If not set, defaults to the value set by <cite>dg plus login</cite>.
Default: `'deployment'`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-organization'>--organization \<organization><a href="#cmdoption-dg-plus-deploy-start-organization" class="hash-link"></a></Link></dt>
<dd>
Dagster+ organization to which to deploy. If not set, defaults to the value set by <cite>dg plus login</cite>.
Default: `'organization'`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-deployment-type'>--deployment-type \<deployment_type_str><a href="#cmdoption-dg-plus-deploy-start-deployment-type" class="hash-link"></a></Link></dt>
<dd>
Whether to deploy to a full deployment or a branch deployment. If unset, will attempt to infer from the current git branch.
Options: full | branch
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-y'>-y, --yes<a href="#cmdoption-dg-plus-deploy-start-y" class="hash-link"></a></Link></dt>
<dd>
Skip confirmation prompts.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-git-url'>--git-url \<git_url><a href="#cmdoption-dg-plus-deploy-start-git-url" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-commit-hash'>--commit-hash \<commit_hash><a href="#cmdoption-dg-plus-deploy-start-commit-hash" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-location-name'>--location-name \<location_names><a href="#cmdoption-dg-plus-deploy-start-location-name" class="hash-link"></a></Link></dt>
<dd>
Name of the code location to set the build output for. Defaults to the current project’s code location, or every project’s code location when run in a workspace.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-deploy-start-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-status-url'>--status-url \<status_url><a href="#cmdoption-dg-plus-deploy-start-status-url" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-snapshot-base-condition'>--snapshot-base-condition \<snapshot_base_condition_str><a href="#cmdoption-dg-plus-deploy-start-snapshot-base-condition" class="hash-link"></a></Link></dt>
<dd>
Options: on-create | on-update
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-skip-validation'>--skip-validation<a href="#cmdoption-dg-plus-deploy-start-skip-validation" class="hash-link"></a></Link></dt>
<dd>
Skip configuration validation checks (not recommended).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-start-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-start-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_DEPLOYMENT</dt>
<dd>
>
Provide a default for [`--deployment`](#cmdoption-dg-plus-deploy-start-deployment)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-plus-deploy-start-organization)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-deploy-status">
#### status
Show status of the current deploy session.
```shell
dg plus deploy status [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-status-output-format'>--output-format \<output_format><a href="#cmdoption-dg-plus-deploy-status-output-format" class="hash-link"></a></Link></dt>
<dd>
Output format for the deploy status.
Options: json | markdown
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-deploy-status-verbose'>--verbose<a href="#cmdoption-dg-plus-deploy-status-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
</div></div>
<div class="section" id="dg-plus-integrations">
### integrations
Commands for managing integrations with Dagster Plus.
```shell
dg plus integrations [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-integrations-dbt">
#### dbt
Commands for managing dbt integrations with Dagster Plus.
```shell
dg plus integrations dbt [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-integrations-dbt-download-manifest">
##### download-manifest
Download a dbt manifest from Dagster Plus for local development.
Downloads the manifest artifact stored at organization scope. Use –components
to discover DbtProject instances from a dg project, or –file to point at a
Python file containing DbtProject definitions.
```shell
dg plus integrations dbt download-manifest [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-components'>--components \<components_path><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-components" class="hash-link"></a></Link></dt>
<dd>
Path to a dg project directory containing DbtProjectComponents.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-file'>--file \<file_path><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-file" class="hash-link"></a></Link></dt>
<dd>
Path to a Python file with DbtProject definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-key-prefix'>--key-prefix \<key_prefix><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-key-prefix" class="hash-link"></a></Link></dt>
<dd>
A key prefix for the key the manifest.json is saved with.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-output'>--output \<output_path><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-output" class="hash-link"></a></Link></dt>
<dd>
Override the download destination path. Cannot be used with multiple projects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-o'>-o, --organization \<organization><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-api-token'>--api-token \<api_token><a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-download-manifest-view-graphql'>--view-graphql<a href="#cmdoption-dg-plus-integrations-dbt-download-manifest-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-plus-integrations-dbt-download-manifest-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-plus-integrations-dbt-download-manifest-api-token)
</dd>
</dl>
</div>
<div class="section" id="dg-plus-integrations-dbt-manage-manifest">
##### manage-manifest
Auto-manage dbt manifest upload/download based on deployment context.
In branch deployments, downloads the prod manifest. In the source deployment
(default: “prod”), uploads the manifest. Replaces <cite>dagster-cloud ci project manage-state</cite>.
```shell
dg plus integrations dbt manage-manifest [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-components'>--components \<components_path><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-components" class="hash-link"></a></Link></dt>
<dd>
Path to a dg project directory containing DbtProjectComponents.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-file'>--file \<file_path><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-file" class="hash-link"></a></Link></dt>
<dd>
Path to a Python file with DbtProject definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-source-deployment'>--source-deployment \<source_deployment><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-source-deployment" class="hash-link"></a></Link></dt>
<dd>
Which deployment should upload its manifest.json.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-key-prefix'>--key-prefix \<key_prefix><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-key-prefix" class="hash-link"></a></Link></dt>
<dd>
A key prefix for the key the manifest.json is saved with.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-o'>-o, --organization \<organization><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-o" class="hash-link"></a></Link></dt>
<dd>
Organization to target.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-api-token'>--api-token \<api_token><a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-api-token" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud API token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-integrations-dbt-manage-manifest-view-graphql'>--view-graphql<a href="#cmdoption-dg-plus-integrations-dbt-manage-manifest-view-graphql" class="hash-link"></a></Link></dt>
<dd>
Print GraphQL queries and responses to stderr for debugging.
</dd>
</dl>
Environment variables:
<dl>
<dt>DAGSTER_CLOUD_ORGANIZATION</dt>
<dd>
>
Provide a default for [`--organization`](#cmdoption-dg-plus-integrations-dbt-manage-manifest-o)
</dd>
</dl>
<dl>
<dt>DAGSTER_CLOUD_API_TOKEN</dt>
<dd>
>
Provide a default for [`--api-token`](#cmdoption-dg-plus-integrations-dbt-manage-manifest-api-token)
</dd>
</dl>
</div></div></div>
<div class="section" id="dg-plus-login">
### login
Login to Dagster Plus.
```shell
dg plus login [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-login-region'>--region \<region><a href="#cmdoption-dg-plus-login-region" class="hash-link"></a></Link></dt>
<dd>
Dagster Cloud region to login to. Use ‘eu’ for European region.
Options: us | eu
</dd>
</dl>
</div>
<div class="section" id="dg-plus-pull">
### pull
Commands for pulling configuration from Dagster Plus.
```shell
dg plus pull [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-plus-pull-env">
#### env
Pull environment variables from Dagster Plus and save to a .env file for local use.
```shell
dg plus pull env [OPTIONS]
```
Example:
```default
$ dg plus pull env
Environment variables saved to .env
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-pull-env-verbose'>--verbose<a href="#cmdoption-dg-plus-pull-env-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-plus-pull-env-target-path'>--target-path \<target_path><a href="#cmdoption-dg-plus-pull-env-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
</div></div></div></div>
---
---
title: 'dg utils reference'
sidebar_position: 1000
title_meta: 'dg utils reference API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dg utils reference Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dg-utils-reference">
# dg utils reference
<div class="section" id="dg-utils">
## dg utils
Assorted utility commands.
```shell
dg utils [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dg-utils-configure-editor">
### configure-editor
Generates and installs a VS Code or Cursor extension which provides JSON schemas for Components types specified by YamlComponentsLoader objects.
```shell
dg utils configure-editor [OPTIONS] {vscode|cursor}
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-configure-editor-target-path'>--target-path \<target_path><a href="#cmdoption-dg-utils-configure-editor-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-configure-editor-verbose'>--verbose<a href="#cmdoption-dg-utils-configure-editor-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-configure-editor-arg-EDITOR'>EDITOR<a href="#cmdoption-dg-utils-configure-editor-arg-EDITOR" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
</div>
<div class="section" id="dg-utils-generate-component-schema">
### generate-component-schema
Generates a JSON schema for the component types installed in the current project.
```shell
dg utils generate-component-schema [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-generate-component-schema-verbose'>--verbose<a href="#cmdoption-dg-utils-generate-component-schema-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-generate-component-schema-output-path'>--output-path \<output_path><a href="#cmdoption-dg-utils-generate-component-schema-output-path" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
<div class="section" id="dg-utils-inspect-component">
### inspect-component
Get detailed information on a registered Dagster component type.
```shell
dg utils inspect-component [OPTIONS] COMPONENT_TYPE
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-description'>--description<a href="#cmdoption-dg-utils-inspect-component-description" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-scaffold-params-schema'>--scaffold-params-schema<a href="#cmdoption-dg-utils-inspect-component-scaffold-params-schema" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-defs-yaml-json-schema'>--defs-yaml-json-schema<a href="#cmdoption-dg-utils-inspect-component-defs-yaml-json-schema" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-defs-yaml-schema'>--defs-yaml-schema<a href="#cmdoption-dg-utils-inspect-component-defs-yaml-schema" class="hash-link"></a></Link></dt>
<dd>
Generate LLM-optimized YAML template with inline documentation and type hints. Unlike JSON schemas designed for validation, this YAML format includes human-readable annotations and structured documentation that LLMs can better parse and understand. Includes Required/Optional annotations, plain English type descriptions, field descriptions inline with properties, and serves as both documentation and code generation scaffold optimized for AI consumption.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-defs-yaml-example-values'>--defs-yaml-example-values<a href="#cmdoption-dg-utils-inspect-component-defs-yaml-example-values" class="hash-link"></a></Link></dt>
<dd>
Generate YAML example values optimized for LLM understanding and code generation
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-target-path'>--target-path \<target_path><a href="#cmdoption-dg-utils-inspect-component-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-verbose'>--verbose<a href="#cmdoption-dg-utils-inspect-component-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-inspect-component-arg-COMPONENT_TYPE'>COMPONENT_TYPE<a href="#cmdoption-dg-utils-inspect-component-arg-COMPONENT_TYPE" class="hash-link"></a></Link></dt>
<dd>
Required argument
</dd>
</dl>
</div>
<div class="section" id="dg-utils-refresh-defs-state">
### refresh-defs-state
Refresh the defs state for the current project.
```shell
dg utils refresh-defs-state [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-refresh-defs-state-target-path'>--target-path \<target_path><a href="#cmdoption-dg-utils-refresh-defs-state-target-path" class="hash-link"></a></Link></dt>
<dd>
Specify a directory to use to load the context for this command. This will typically be a folder with a dg.toml or pyproject.toml file in it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-refresh-defs-state-verbose'>--verbose<a href="#cmdoption-dg-utils-refresh-defs-state-verbose" class="hash-link"></a></Link></dt>
<dd>
Enable verbose output for debugging.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-refresh-defs-state-defs-state-key'>--defs-state-key \<defs_state_key><a href="#cmdoption-dg-utils-refresh-defs-state-defs-state-key" class="hash-link"></a></Link></dt>
<dd>
Only refresh state for specified defs state key. Can be specified multiple times.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dg-utils-refresh-defs-state-management-type'>--management-type \<management_type><a href="#cmdoption-dg-utils-refresh-defs-state-management-type" class="hash-link"></a></Link></dt>
<dd>
Only refresh components with the specified management type. Can be specified multiple times to include multiple types. Defaults to all management types except for LEGACY_CODE_SERVER_SNAPSHOTS.
Options: LOCAL_FILESYSTEM | VERSIONED_STATE_STORAGE
</dd>
</dl>
</div></div></div>
---
---
title: 'asset checks'
sidebar_position: 1000
title_meta: 'asset checks API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'asset checks Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="asset-checks">
# Asset checks
Dagster allows you to define and execute checks on your software-defined assets. Each asset check verifies some property of a data asset, e.g. that is has no null values in a particular column.
<dl>
<dt><Link class="anchor" id='dagster.asset_check'>@dagster.asset_check <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_check_decorator.py#L101' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.asset_check" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to execute an asset check.
Parameters:
- <strong>asset</strong> (<em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em>) – The asset that the check applies to.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the check. If not specified, the name of the decorated function will be used. Checks for the same asset must have unique names.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The description of the check.
- <strong>blocking</strong> (<em>bool</em>) – When enabled, runs that include this check and any downstream assets that depend on <cite>asset</cite> will wait for this check to complete before starting the downstream assets. If the check fails with severity <cite>AssetCheckSeverity.ERROR</cite>, then the downstream assets won’t execute.
- <strong>additional_ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](assets.mdx#dagster.AssetIn)<em>]</em><em>]</em>) – A mapping from input name to information about the input. These inputs will apply to the underlying op that executes the check. These should not include the <cite>asset</cite> parameter, which is always included as a dependency.
- <strong>additional_deps</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em><em>CoercibleToAssetDep</em><em>]</em><em>]</em>) – Assets that are upstream dependencies, but do not correspond to a parameter of the decorated function. These dependencies will apply to the underlying op that executes the check. These should not include the <cite>asset</cite> parameter, which is always included as a dependency.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of keys for resources that are required by the function that execute the check. These can alternatively be specified by including resource-typed parameters in the function signature.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The configuration schema for the check’s underlying op. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that executes the check. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>compute_kind</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string to represent the kind of computation that executes the check, e.g. “dbt” or “spark”.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that executes the check.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of static metadata for the check.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](assets.mdx#dagster.AutomationCondition)<em>]</em>) – An AutomationCondition which determines when this check should be executed.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs this asset check’s execution.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – The PartitionsDefinition for this asset check.
Produces an [`AssetChecksDefinition`](#dagster.AssetChecksDefinition) object.
Example:
```python
from dagster import asset, asset_check, AssetCheckResult
@asset
def my_asset() -> None:
...
@asset_check(asset=my_asset, description="Check that my asset has enough rows")
def my_asset_has_enough_rows() -> AssetCheckResult:
num_rows = ...
return AssetCheckResult(passed=num_rows > 5, metadata={"num_rows": num_rows})
```
Example with a DataFrame Output:
```python
from dagster import asset, asset_check, AssetCheckResult
from pandas import DataFrame
@asset
def my_asset() -> DataFrame:
...
@asset_check(asset=my_asset, description="Check that my asset has enough rows")
def my_asset_has_enough_rows(my_asset: DataFrame) -> AssetCheckResult:
num_rows = my_asset.shape[0]
return AssetCheckResult(passed=num_rows > 5, metadata={"num_rows": num_rows})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckResult'>`class` dagster.AssetCheckResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_result.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckResult" class="hash-link"></a></Link></dt>
<dd>
The result of an asset check.
Parameters:
- <strong>asset_key</strong> (<em>Optional</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – The asset key that was checked.
- <strong>check_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the check.
- <strong>passed</strong> (<em>bool</em>) – The pass/fail result of the check.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the asset. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
- <strong>severity</strong> ([*AssetCheckSeverity*](#dagster.AssetCheckSeverity)) – Severity of the check. Defaults to ERROR.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A text description of the result of the check evaluation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckSpec'>`class` dagster.AssetCheckSpec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_spec.py#L58' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckSpec" class="hash-link"></a></Link></dt>
<dd>
Defines information about an asset check, except how to execute it.
AssetCheckSpec is often used as an argument to decorators that decorator a function that can
execute multiple checks - e.g. <cite>@asset</cite>, and <cite>@multi_asset</cite>. It defines one of the checks that
will be executed inside that function.
Parameters:
- <strong>name</strong> (<em>str</em>) – Name of the check.
- <strong>asset</strong> (<em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em>) – The asset that the check applies to.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Description for the check.
- <strong>additional_deps</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em>[*AssetDep*](assets.mdx#dagster.AssetDep)<em>]</em><em>]</em>) – Additional dependencies for the check. The check relies on these assets in some way, but the result of the check only applies to the asset specified by <cite>asset</cite>. For example, the check may test that <cite>asset</cite> has matching data with an asset in <cite>additional_deps</cite>. This field holds both <cite>additional_deps</cite> and <cite>additional_ins</cite> passed to @asset_check.
- <strong>blocking</strong> (<em>bool</em>) – When enabled, if the check fails with severity <cite>AssetCheckSeverity.ERROR</cite>, then downstream assets won’t execute. If this AssetCheckSpec is used in a multi-asset, that multi-asset is responsible for enforcing that downstream assets within the same step do not execute after a blocking asset check fails. Note that gating applies only to <em>failed</em> check results; if no result is emitted for the check, downstream execution proceeds and a warning is logged.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of static metadata for this asset check.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](assets.mdx#dagster.AutomationCondition)<em>[</em>[*AssetCheckKey*](#dagster.AssetCheckKey)<em>]</em><em>]</em>) – The AutomationCondition for this asset check.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – The PartitionsDefinition for this asset check. Must be either None or the same as the PartitionsDefinition of the asset specified by <cite>asset</cite>.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckSeverity'>`class` dagster.AssetCheckSeverity <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_spec.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckSeverity" class="hash-link"></a></Link></dt>
<dd>
Severity level for an AssetCheckResult.
- WARN: a potential issue with the asset
- ERROR: a definite issue with the asset
Severity does not impact execution of the asset or downstream assets.
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckSeverity.ERROR'>ERROR `=` 'ERROR'<a href="#dagster.AssetCheckSeverity.ERROR" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckSeverity.WARN'>WARN `=` 'WARN'<a href="#dagster.AssetCheckSeverity.WARN" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckKey'>`class` dagster.AssetCheckKey <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_key.py#L211' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckKey" class="hash-link"></a></Link></dt>
<dd>
Check names are expected to be unique per-asset. Thus, this combination of asset key and
check name uniquely identifies an asset check within a deployment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.multi_asset_check'>@dagster.multi_asset_check <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_check_decorator.py#L292' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multi_asset_check" class="hash-link"></a></Link></dt>
<dd>
Defines a set of asset checks that can be executed together with the same op.
Parameters:
- <strong>specs</strong> (<em>Sequence</em><em>[</em>[*AssetCheckSpec*](#dagster.AssetCheckSpec)<em>]</em>) – Specs for the asset checks.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op. If not specified, the name of the decorated function will be used.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Description of the op.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of keys for resources that are required by the function that execute the checks. These can alternatively be specified by including resource-typed parameters in the function signature.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The configuration schema for the asset checks’ underlying op. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that executes the checks. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>compute_kind</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string to represent the kind of computation that executes the checks, e.g. “dbt” or “spark”.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that executes the checks.
- <strong>can_subset</strong> (<em>bool</em>) – Whether the op can emit results for a subset of the asset checks keys, based on the context.selected_asset_check_keys argument. Defaults to False.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em>[*AssetIn*](assets.mdx#dagster.AssetIn)<em>]</em><em>]</em><em>]</em>) – A mapping from input name to AssetIn depended upon by a given asset check. If an AssetKey is provided, it will be converted to an AssetIn with the same key.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs this multi asset check’s execution.
Examples:
```python
@multi_asset_check(
specs=[
AssetCheckSpec("enough_rows", asset="asset1"),
AssetCheckSpec("no_dupes", asset="asset1"),
AssetCheckSpec("enough_rows", asset="asset2"),
],
)
def checks():
yield AssetCheckResult(passed=True, asset_key="asset1", check_name="enough_rows")
yield AssetCheckResult(passed=False, asset_key="asset1", check_name="no_dupes")
yield AssetCheckResult(passed=True, asset_key="asset2", check_name="enough_rows")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_asset_checks_from_modules'>dagster.load_asset_checks_from_modules <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_asset_checks_from_modules.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_asset_checks_from_modules" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of asset checks from the given modules. This is most often used in
conjunction with a call to <cite>load_assets_from_modules</cite>.
Parameters:
- <strong>modules</strong> (<em>Iterable</em><em>[</em><em>ModuleType</em><em>]</em>) – The Python modules to look for checks inside.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – The prefix for the asset keys targeted by the loaded checks. This should match the key_prefix argument to load_assets_from_modules.
Returns: A list containing asset checks defined in the given modules.Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_asset_checks_from_current_module'>dagster.load_asset_checks_from_current_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_asset_checks_from_modules.py#L52' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_asset_checks_from_current_module" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of asset checks from the module where this function is called. This is most
often used in conjunction with a call to <cite>load_assets_from_current_module</cite>.
Parameters: <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – The prefix for the asset keys targeted by the loaded checks. This should match the
key_prefix argument to load_assets_from_current_module.Returns: A list containing asset checks defined in the current module.Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_asset_checks_from_package_module'>dagster.load_asset_checks_from_package_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_asset_checks_from_modules.py#L79' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_asset_checks_from_package_module" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of asset checks from all sub-modules of the given package module. This is
most often used in conjunction with a call to <cite>load_assets_from_package_module</cite>.
Parameters:
- <strong>package_module</strong> (<em>ModuleType</em>) – The Python module to look for checks inside.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – The prefix for the asset keys targeted by the loaded checks. This should match the key_prefix argument to load_assets_from_package_module.
Returns: A list containing asset checks defined in the package.Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_asset_checks_from_package_name'>dagster.load_asset_checks_from_package_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_asset_checks_from_modules.py#L104' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_asset_checks_from_package_name" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of asset checks from all sub-modules of the given package. This is most
often used in conjunction with a call to <cite>load_assets_from_package_name</cite>.
Parameters:
- <strong>package_name</strong> (<em>str</em>) – The name of the Python package to look for checks inside.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – The prefix for the asset keys targeted by the loaded checks. This should match the key_prefix argument to load_assets_from_package_name.
Returns: A list containing asset checks defined in the package.Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetChecksDefinition'>`class` dagster.AssetChecksDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L410' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetChecksDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a set of checks that are produced by the same op or op graph.
AssetChecksDefinition should not be instantiated directly, but rather produced using the <cite>@asset_check</cite> decorator or <cite>AssetChecksDefinition.create</cite> method.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_last_update_freshness_checks'>dagster.build_last_update_freshness_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/freshness_checks/last_update.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_last_update_freshness_checks" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Attach `FreshnessPolicy` objects to your assets instead..
:::
Constructs an <cite>AssetChecksDefinition</cite> that checks the freshness of the provided assets.
This check passes if the asset is found to be “fresh”, and fails if the asset is found to be
“overdue”. An asset is considered fresh if a record (i.e. a materialization or observation)
exists with a timestamp greater than the “lower bound” derived from the parameters of this
function.
<cite>deadline_cron</cite> is a cron schedule that defines the deadline for when we should expect the asset
to arrive by; if not provided, we consider the deadline to be the execution time of the check.
<cite>lower_bound_delta</cite> is a timedelta that defines the lower bound for when a record could have
arrived by. If the most recent recent record’s timestamp is earlier than
<cite>deadline-lower_bound_delta</cite>, the asset is considered overdue.
Let’s use two examples, one with a deadline_cron set and one without.
Let’s say I have an asset which runs on a schedule every day at 8:00 AM UTC, and usually takes
around 45 minutes to complete. To account for operational delays, I would expect the asset to be
done materializing every day by 9:00 AM UTC. I would set the <cite>deadline_cron</cite> to “0 9 * * *”, and
the <cite>lower_bound_delta</cite> to “45 minutes”. This would mean that starting at 9:00 AM, this check
will expect a materialization record to have been created no earlier than 8:15 AM. Note that if
the check runs at 8:59 AM, the deadline has not yet passed, and we’ll instead be checking for
the most recently passed deadline, which is yesterday.
Let’s say I have an observable source asset on a data source which I expect should never be more
than 3 hours out of date. In this case, there’s no fixed schedule for when the data should be
updated, so I would not provide a <cite>deadline_cron</cite>. Instead, I would set the <cite>lower_bound_delta</cite>
parameter to “3 hours”. This would mean that the check will expect the most recent observation
record to indicate data no older than 3 hours, relative to the current time, regardless of when it runs.
The check result will contain the following metadata:
“dagster/freshness_params”: A dictionary containing the parameters used to construct the
check
“dagster/last_updated_time”: The time of the most recent update to the asset
“dagster/overdue_seconds”: (Only present if asset is overdue) The number of seconds that the
asset is overdue by.
“dagster/overdue_deadline_timestamp”: The timestamp that we are expecting the asset to have
arrived by. In the case of a provided deadline_cron, this is the timestamp of the most recent
tick of the cron schedule. In the case of no deadline_cron, this is the current time.
Examples:
```python
# Example 1: Assets that are expected to be updated every day within 45 minutes of
# 9:00 AM UTC
from dagster import build_last_update_freshness_checks, AssetKey
from .somewhere import my_daily_scheduled_assets_def
checks_def = build_last_update_freshness_checks(
[my_daily_scheduled_assets_def, AssetKey("my_other_daily_asset_key")],
lower_bound_delta=datetime.timedelta(minutes=45),
deadline_cron="0 9 * * *",
)
# Example 2: Assets that are expected to be updated within 3 hours of the current time
from dagster import build_last_update_freshness_checks, AssetKey
from .somewhere import my_observable_source_asset
checks_def = build_last_update_freshness_checks(
[my_observable_source_asset, AssetKey("my_other_observable_asset_key")],
lower_bound_delta=datetime.timedelta(hours=3),
)
```
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetKey</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em>) – The assets to construct checks for. All checks are incorporated into the same <cite>AssetChecksDefinition</cite>, which can be subsetted to run checks for specific assets.
- <strong>lower_bound_delta</strong> (<em>datetime.timedelta</em>) – The check will pass if the asset was updated within lower_bound_delta of the current_time (no cron) or the most recent tick of the cron (cron provided).
- <strong>deadline_cron</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Defines the deadline for when we should start checking that the asset arrived. If not provided, the deadline is the execution time of the check.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The timezone to use when calculating freshness and deadline. If not provided, defaults to “UTC”.
- <strong>blocking</strong> (<em>bool</em>) – Whether the check should block execution if it fails. Defaults to False.
Returns:
<cite>AssetChecksDefinition</cite> objects which execute freshness checks
for the provided assets.
Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_time_partition_freshness_checks'>dagster.build_time_partition_freshness_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/freshness_checks/time_partition.py#L45' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_time_partition_freshness_checks" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Attach `FreshnessPolicy` objects to your assets instead..
:::
Construct an <cite>AssetChecksDefinition</cite> that checks the freshness of the provided assets.
This check passes if the asset is considered “fresh” by the time that execution begins. We
consider an asset to be “fresh” if there exists a record for the most recent partition, once
the deadline has passed.
<cite>deadline_cron</cite> is a cron schedule that defines the deadline for when we should expect the most
recent partition to arrive by. Once a tick of the cron schedule has passed, this check will fail
if the most recent partition has not been observed/materialized.
Let’s say I have a daily-partitioned asset which runs every day at 8:00 AM UTC, and takes around
45 minutes to complete. To account for operational delays, I would expect the asset to be done
materializing every day by 9:00 AM UTC. I would set the <cite>deadline_cron</cite> to “0 9 * * *”. This
means that starting at 9:00 AM, this check will expect a record to exist for the previous day’s
partition. Note that if the check runs at 8:59 AM, the deadline has not yet passed, and we’ll
instead be checking for the most recently passed deadline, which is yesterday (meaning the
partition representing the day before yesterday).
The timestamp of an observation record is the timestamp indicated by the
“dagster/last_updated_timestamp” metadata key. The timestamp of a materialization record is the
timestamp at which that record was created.
The check will fail at runtime if a non-time-window partitioned asset is passed in.
The check result will contain the following metadata:
“dagster/freshness_params”: A dictionary containing the parameters used to construct the
check.
“dagster/last_updated_time”: (Only present if the asset has been observed/materialized before)
The time of the most recent update to the asset.
“dagster/overdue_seconds”: (Only present if asset is overdue) The number of seconds that the
asset is overdue by.
“dagster/overdue_deadline_timestamp”: The timestamp that we are expecting the asset to have
arrived by. This is the timestamp of the most recent tick of the cron schedule.
Examples:
```python
from dagster import build_time_partition_freshness_checks, AssetKey
# A daily partitioned asset that is expected to be updated every day within 45 minutes
# of 9:00 AM UTC
from .somewhere import my_daily_scheduled_assets_def
checks_def = build_time_partition_freshness_checks(
[my_daily_scheduled_assets_def],
deadline_cron="0 9 * * *",
)
```
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetKey</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em>) – The assets to construct checks for. For each passed in asset, there will be a corresponding constructed <cite>AssetChecksDefinition</cite>.
- <strong>deadline_cron</strong> (<em>str</em>) – The check will pass if the partition time window most recently completed by the time of the last cron tick has been observed/materialized.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The timezone to use when calculating freshness and deadline. If not provided, defaults to “UTC”.
- <strong>severity</strong> ([*AssetCheckSeverity*](#dagster.AssetCheckSeverity)) – The severity of the check. Defaults to “ERROR”.
- <strong>blocking</strong> (<em>bool</em>) – Whether the check should block execution if it fails. Defaults to False.
Returns:
<cite>AssetChecksDefinition</cite> objects which execute freshness
checks for the provided assets.
Return type: Sequence[[AssetChecksDefinition](#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_sensor_for_freshness_checks'>dagster.build_sensor_for_freshness_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/freshness_checks/sensor.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_sensor_for_freshness_checks" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `FreshnessPolicy` objects, which do not require a sensor, instead..
:::
Builds a sensor which kicks off evaluation of freshness checks.
This sensor will kick off an execution of a check in the following cases:
- The check has never been executed before.
- The check has been executed before, and the previous result was a success, but it is again
possible for the check to be overdue based on the <cite>dagster/fresh_until_timestamp</cite> metadata
on the check result.
Note that we will not execute if:
- The freshness check has been executed before, and the previous result was a failure. This is
because whichever run materializes/observes the run to bring the check back to a passing
state will end up also running the check anyway, so until that run occurs, there’s no point
in evaluating the check.
- The freshness check has been executed before, and the previous result was a success, but it is
not possible for the check to be overdue based on the <cite>dagster/fresh_until_timestamp</cite>
metadata on the check result. Since the check cannot be overdue, we know the check
result would not change with an additional execution.
Parameters:
- <strong>freshness_checks</strong> (<em>Sequence</em><em>[</em>[*AssetChecksDefinition*](#dagster.AssetChecksDefinition)<em>]</em>) – The freshness checks to evaluate.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The duration in seconds between evaluations of the sensor.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to “freshness_check_sensor”, but a name may need to be provided in case of multiple calls of this function.
- <strong>default_status</strong> (<em>Optional</em><em>[</em><em>DefaultSensorStatus</em><em>]</em>) – The default status of the sensor. Defaults to stopped.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags (string key-value pairs) to attach to the launched run.
Returns: The sensor that kicks off freshness evaluations.Return type: [SensorDefinition](schedules-sensors.mdx#dagster.SensorDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_column_schema_change_checks'>dagster.build_column_schema_change_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/schema_change_checks.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_column_schema_change_checks" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns asset checks that pass if the column schema of the asset’s latest materialization
is the same as the column schema of the asset’s previous materialization.
The underlying materializations are expected to have a metadata entry with key <cite>dagster/column_schema</cite> and type [`TableSchema`](metadata.mdx#dagster.TableSchema).
To learn more about how to add column schema metadata and other forms of tabular metadata to assets, see
[https://docs.dagster.io/guides/build/assets/metadata-and-tags/table-metadata#attaching-column-schema](https://docs.dagster.io/guides/build/assets/metadata-and-tags/table-metadata#attaching-column-schema).
The resulting checks will fail if any changes are detected in the column schema between
materializations, including:
- Added columns
- Removed columns
- Changes to column types
The check failure message will detail exactly what changed in the schema.
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>str</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em>) – The assets to create asset checks for.
- <strong>severity</strong> ([*AssetCheckSeverity*](#dagster.AssetCheckSeverity)) – The severity if the check fails. Defaults to WARN.
Returns: Sequence[AssetsChecksDefinition]
Examples:
First, define an asset with column schema metadata. You can attach schema metadata either as
definition metadata (when schema is known at definition time) or as materialization metadata
(when schema is only known at runtime):
```python
import dagster as dg
# Using definition metadata when schema is known upfront
@dg.asset
def people_table():
column_names = ...
column_types = ...
columns = [
dg.TableColumn(name, column_type)
for name, column_type in zip(column_names, column_types)
]
yield dg.MaterializeResult(
metadata={"dagster/column_schema": dg.TableSchema(columns=columns)}
)
```
Once you have assets with column schema metadata, you can create schema change checks to monitor
for changes in the schema between materializations:
```python
# Create schema change checks for one or more assets
schema_checks = dg.build_column_schema_change_checks(
assets=[people_table]
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_metadata_bounds_checks'>dagster.build_metadata_bounds_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_checks/asset_check_factories/metadata_bounds_checks.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_metadata_bounds_checks" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns asset checks that pass if the metadata value of the asset’s latest materialization
is within the specified range.
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>str</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em>) – The assets to create asset checks for.
- <strong>severity</strong> ([*AssetCheckSeverity*](#dagster.AssetCheckSeverity)) – The severity if the check fails. Defaults to WARN.
- <strong>metadata_key</strong> (<em>str</em>) – The metadata key to check.
- <strong>min_value</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>int</em><em>, </em><em>float</em><em>]</em><em>]</em>) – The minimum value to check for. If None, no minimum value check is performed.
- <strong>max_value</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>int</em><em>, </em><em>float</em><em>]</em><em>]</em>) – The maximum value to check for. If None, no maximum value check is performed.
- <strong>exclusive_min</strong> (<em>bool</em>) – If True, the check will fail if the metadata value is equal to <cite>min_value</cite>. Defaults to False.
- <strong>exclusive_max</strong> (<em>bool</em>) – If True, the check will fail if the metadata value is equal to <cite>max_value</cite>. Defaults to False.
Returns: Sequence[AssetsChecksDefinition]
</dd>
</dl>
</div>
---
---
title: 'assets'
sidebar_position: 1000
title_meta: 'assets API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'assets Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="assets">
# Assets
An asset is an object in persistent storage, such as a table, file, or persisted machine learning model. An asset definition is a description, in code, of an asset that should exist and how to produce and update that asset.
<div class="section" id="asset-definitions">
## Asset definitions
Refer to the [Asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.asset'>@dagster.asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.asset" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to compute an asset.
A software-defined asset is the combination of:
1. An asset key, e.g. the name of a table.
2. A function, which can be run to compute the contents of the asset.
3. A set of upstream assets that are provided as inputs to the function when computing the asset.
Unlike an op, whose dependencies are determined by the graph it lives inside, an asset knows
about the upstream assets it depends on. The upstream assets are inferred from the arguments
to the decorated function. The name of the argument designates the name of the upstream asset.
An asset has an op inside it to represent the function that computes it. The name of the op
will be the segments of the asset key, separated by double-underscores.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the asset. If not provided, defaults to the name of the decorated function. The asset’s name must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the asset’s key is the concatenation of the key_prefix and the asset’s name, which defaults to the name of the decorated function. Each item in key_prefix must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](#dagster.AssetIn)<em>]</em><em>]</em>) – A dictionary that maps input names to information about the input.
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetDep*](#dagster.AssetDep)<em>, </em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](#dagster.SourceAsset)<em>, </em>[*AssetKey*](#dagster.AssetKey)<em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – The assets that are upstream dependencies, but do not correspond to a parameter of the decorated function. If the AssetsDefinition for a multi_asset is provided, dependencies on all assets created by the multi_asset will be created.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The configuration schema for the asset’s underlying op. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of metadata entries for the asset.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource handles required by the op.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The resource key of the IOManager used for storing the output of the op as an asset, and for loading it in downstream ops (default: “io_manager”). Only one of io_manager_key and io_manager_def can be provided.
- <strong>io_manager_def</strong> (<em>Optional</em><em>[</em><em>object</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) The IOManager used for storing the output of the op as an asset, and for loading it in downstream ops. Only one of io_manager_def and io_manager_key can be provided.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em>) – Allows specifying type validation functions that will be executed on the output of the decorated function after it runs.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A mapping of resource keys to resources. These resources will be initialized during execution, and can be accessed from the context within the body of the function.
- <strong>hooks</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A set of hooks to attach to the asset. These hooks will be executed when the asset is materialized.
- <strong>output_required</strong> (<em>bool</em>) – Whether the decorated function will always materialize an asset. Defaults to True. If False, the function can conditionally not <cite>yield</cite> a result. If no result is yielded, no output will be materialized to storage and downstream assets will not be materialized. Note that for <cite>output_required</cite> to work at all, you must use <cite>yield</cite> in your asset logic rather than <cite>return</cite>. <cite>return</cite> will not respect this setting and will always produce an asset materialization, even if <cite>None</cite> is returned.
- <strong>automation_condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – A condition describing when Dagster should materialize this asset.
- <strong>backfill_policy</strong> ([*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)) – <span className="flag flag-info">beta</span> (Beta) Configure Dagster to backfill this asset according to its BackfillPolicy.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that computes the asset.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the code that generates this asset. In general, versions should be set only for code that deterministically produces the same output when given the same inputs.
- <strong>check_specs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckSpec*](asset-checks.mdx#dagster.AssetCheckSpec)<em>]</em><em>]</em>) – Specs for asset checks that execute in the decorated function after materializing the asset.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>CoeercibleToAssetKey</em><em>]</em>) – The key for this asset. If provided, cannot specify key_prefix or name.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing owners of the asset. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
- <strong>kinds</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing the kinds of the asset. These will be made visible in the Dagster UI.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs this asset’s execution.
- <strong>non_argument_deps</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Set</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>, </em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> Deprecated, use deps instead. Set of asset keys that are upstream dependencies, but do not pass an input to the asset. Hidden parameter not exposed in the decorator signature, but passed in kwargs.
Examples:
```python
@asset
def my_upstream_asset() -> int:
return 5
@asset
def my_asset(my_upstream_asset: int) -> int:
return my_upstream_asset + 1
should_materialize = True
@asset(output_required=False)
def conditional_asset():
if should_materialize:
yield Output(5) # you must `yield`, not `return`, the result
# Will also only materialize if `should_materialize` is `True`
@asset
def downstream_asset(conditional_asset):
return conditional_asset + 1
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MaterializeResult'>`class` dagster.MaterializeResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/result.py#L62' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MaterializeResult" class="hash-link"></a></Link></dt>
<dd>
An object representing a successful materialization of an asset. These can be returned from
@asset and @multi_asset decorated functions to pass metadata or specify specific assets were
materialized.
Parameters:
- <strong>asset_key</strong> (<em>Optional</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – Optional in @asset, required in @multi_asset to discern which asset this refers to.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em>) – Metadata to record with the corresponding AssetMaterialization event.
- <strong>check_results</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckResult*](asset-checks.mdx#dagster.AssetCheckResult)<em>]</em><em>]</em>) – Check results to record with the corresponding AssetMaterialization event.
- <strong>data_version</strong> (<em>Optional</em><em>[</em><em>DataVersion</em><em>]</em>) – The data version of the asset that was observed.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags to record with the corresponding AssetMaterialization event.
- <strong>value</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The output value of the asset that was materialized.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSpec'>`class` dagster.AssetSpec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_spec.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSpec" class="hash-link"></a></Link></dt>
<dd>
Specifies the core attributes of an asset, except for the function that materializes or
observes it.
An asset spec plus any materialization or observation function for the asset constitutes an
“asset definition”.
Parameters:
- <strong>key</strong> ([*AssetKey*](#dagster.AssetKey)) – The unique identifier for this asset.
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – The asset keys for the upstream assets that materializing this asset depends on.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of this asset.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of static metadata for this asset. For example, users can provide information about the database table this asset corresponds to.
- <strong>skippable</strong> (<em>bool</em>) – Whether this asset can be omitted during materialization, causing downstream dependencies to skip.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The version of the code for this specific asset, overriding the code version of the materialization function
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to the specified asset.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing owners of the asset. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – The automation condition to apply to the asset.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
- <strong>kinds</strong> – (Optional[Set[str]]): A set of strings representing the kinds of the asset. These will be made visible in the Dagster UI.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
<dl>
<dt><Link class="anchor" id='dagster.AssetSpec.merge_attributes'>merge_attributes <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_spec.py#L363' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSpec.merge_attributes" class="hash-link"></a></Link></dt>
<dd>
Returns a new AssetSpec with the specified attributes merged with the current attributes.
Parameters:
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em><em>CoercibleToAssetDep</em><em>]</em><em>]</em>) – A set of asset dependencies to add to the asset self.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A set of metadata to add to the asset self. Will overwrite any existing metadata with the same key.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of owners to add to the asset self.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of tags to add to the asset self. Will overwrite any existing tags with the same key.
- <strong>kinds</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of kinds to add to the asset self.
Returns: AssetSpec
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSpec.replace_attributes'>replace_attributes <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_spec.py#L314' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSpec.replace_attributes" class="hash-link"></a></Link></dt>
<dd>
Returns a new AssetSpec with the specified attributes replaced.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSpec.with_io_manager_key'>with_io_manager_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_spec.py#L298' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSpec.with_io_manager_key" class="hash-link"></a></Link></dt>
<dd>
Returns a copy of this AssetSpec with an extra metadata value that dictates which I/O
manager to use to load the contents of this asset in downstream computations.
Parameters: <strong>io_manager_key</strong> (<em>str</em>) – The I/O manager key. This will be used as the value for the
“dagster/io_manager_key” metadata key.Returns: AssetSpec
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition'>`class` dagster.AssetsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L410' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a set of assets that are produced by the same op or graph.
AssetsDefinitions are typically not instantiated directly, but rather produced using the
[`@asset`](#dagster.asset) or [`@multi_asset`](#dagster.multi_asset) decorators.
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.from_graph'>`static` from_graph <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L423' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.from_graph" class="hash-link"></a></Link></dt>
<dd>
Constructs an AssetsDefinition from a GraphDefinition.
Parameters:
- <strong>graph_def</strong> ([*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)) – The GraphDefinition that is an asset.
- <strong>keys_by_input_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – A mapping of the input names of the decorated graph to their corresponding asset keys. If not provided, the input asset keys will be created from the graph input names.
- <strong>keys_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – A mapping of the output names of the decorated graph to their corresponding asset keys. If not provided, the output asset keys will be created from the graph output names.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, key_prefix will be prepended to each key in keys_by_output_name. Each item in key_prefix must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>internal_asset_deps</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Set</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em><em>]</em>) – By default, it is assumed that all assets produced by the graph depend on all assets that are consumed by that graph. If this default is not correct, you pass in a map of output names to a corrected set of AssetKeys that they depend on. Any AssetKeys in this list must be either used as input to the asset or produced within the graph.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the assets.
- <strong>partition_mappings</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*PartitionMapping*](partitions.mdx#dagster.PartitionMapping)<em>]</em><em>]</em>) – Defines how to map partition keys for this asset to partition keys of upstream assets. Each key in the dictionary correponds to one of the input assets, and each value is a PartitionMapping. If no entry is provided for a particular asset dependency, the partition mapping defaults to the default partition mapping for the partitions definition, which is typically maps partition keys to the same partition keys in upstream assets.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A mapping of resource keys to resource definitions. These resources will be initialized during execution, and can be accessed from the body of ops in the graph during execution.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A group name for the constructed asset. Assets without a group name are assigned to a group called “default”.
- <strong>group_names_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Defines a group name to be associated with some or all of the output assets for this node. Keys are names of the outputs, and values are the group name. Cannot be used with the group_name argument.
- <strong>descriptions_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Defines a description to be associated with each of the output asstes for this graph.
- <strong>metadata_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em><em>]</em><em>]</em>) – Defines metadata to be associated with each of the output assets for this node. Keys are names of the outputs, and values are dictionaries of metadata to be associated with the related asset.
- <strong>tags_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – Defines tags to be associated with each of the output assets for this node. Keys are the names of outputs, and values are dictionaries of tags to be associated with the related asset.
- <strong>automation_conditions_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em><em>]</em><em>]</em>) – Defines an AutomationCondition to be associated with some or all of the output assets for this node. Keys are the names of the outputs, and values are the AutoMaterializePolicies to be attached to the associated asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – Defines this asset’s BackfillPolicy
- <strong>owners_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Defines the owners to be associated with each of the output assets for this node. Keys are names of the outputs, and values are sequences of owner strings (user emails or team names).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.from_op'>`static` from_op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L535' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.from_op" class="hash-link"></a></Link></dt>
<dd>
Constructs an AssetsDefinition from an OpDefinition.
Parameters:
- <strong>op_def</strong> ([*OpDefinition*](ops.mdx#dagster.OpDefinition)) – The OpDefinition that is an asset.
- <strong>keys_by_input_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – A mapping of the input names of the decorated op to their corresponding asset keys. If not provided, the input asset keys will be created from the op input names.
- <strong>keys_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – A mapping of the output names of the decorated op to their corresponding asset keys. If not provided, the output asset keys will be created from the op output names.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, key_prefix will be prepended to each key in keys_by_output_name. Each item in key_prefix must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>internal_asset_deps</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Set</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em><em>]</em>) – By default, it is assumed that all assets produced by the op depend on all assets that are consumed by that op. If this default is not correct, you pass in a map of output names to a corrected set of AssetKeys that they depend on. Any AssetKeys in this list must be either used as input to the asset or produced within the op.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the assets.
- <strong>partition_mappings</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*PartitionMapping*](partitions.mdx#dagster.PartitionMapping)<em>]</em><em>]</em>) – Defines how to map partition keys for this asset to partition keys of upstream assets. Each key in the dictionary correponds to one of the input assets, and each value is a PartitionMapping. If no entry is provided for a particular asset dependency, the partition mapping defaults to the default partition mapping for the partitions definition, which is typically maps partition keys to the same partition keys in upstream assets.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A group name for the constructed asset. Assets without a group name are assigned to a group called “default”.
- <strong>group_names_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Defines a group name to be associated with some or all of the output assets for this node. Keys are names of the outputs, and values are the group name. Cannot be used with the group_name argument.
- <strong>descriptions_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Defines a description to be associated with each of the output asstes for this graph.
- <strong>metadata_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em><em>]</em><em>]</em>) – Defines metadata to be associated with each of the output assets for this node. Keys are names of the outputs, and values are dictionaries of metadata to be associated with the related asset.
- <strong>tags_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – Defines tags to be associated with each othe output assets for this node. Keys are the names of outputs, and values are dictionaries of tags to be associated with the related asset.
- <strong>automation_conditions_by_output_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em><em>]</em><em>]</em>) – Defines an AutomationCondition to be associated with some or all of the output assets for this node. Keys are the names of the outputs, and values are the AutoMaterializePolicies to be attached to the associated asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – Defines this asset’s BackfillPolicy
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1490' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Returns a representation of this asset as an [`AssetSpec`](#dagster.AssetSpec).
If this is a multi-asset, the “key” argument allows selecting which asset to return the
spec for.
Parameters: <strong>key</strong> (<em>Optional</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – If this is a multi-asset, select which asset to return its
AssetSpec. If not a multi-asset, this can be left as None.Returns: AssetSpec
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.get_partition_mapping'>get_partition_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1112' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.get_partition_mapping" class="hash-link"></a></Link></dt>
<dd>
Returns the partition mapping between keys in this AssetsDefinition and a given input
asset key (if any).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.to_source_asset'>to_source_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1426' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.to_source_asset" class="hash-link"></a></Link></dt>
<dd>
Returns a representation of this asset as a [`SourceAsset`](#dagster.SourceAsset).
If this is a multi-asset, the “key” argument allows selecting which asset to return a
SourceAsset representation of.
Parameters: <strong>key</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em><em>]</em>) – If this is a multi-asset, select
which asset to return a SourceAsset representation of. If not a multi-asset, this
can be left as None.Returns: SourceAsset
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.to_source_assets'>to_source_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1414' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.to_source_assets" class="hash-link"></a></Link></dt>
<dd>
Returns a SourceAsset for each asset in this definition.
Each produced SourceAsset will have the same key, metadata, io_manager_key, etc. as the
corresponding asset
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.with_hooks'>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1600' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.with_hooks" class="hash-link"></a></Link></dt>
<dd>
Apply a set of hooks to all op instances within the asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.asset_deps'>`property` asset_deps <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L848' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.asset_deps" class="hash-link"></a></Link></dt>
<dd>
Maps assets that are produced by this definition to assets that they depend on. The
dependencies can be either “internal”, meaning that they refer to other assets that are
produced by this definition, or “external”, meaning that they refer to assets that aren’t
produced by this definition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.can_subset'>`property` can_subset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L784' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.can_subset" class="hash-link"></a></Link></dt>
<dd>
If True, indicates that this AssetsDefinition may materialize any subset of its
asset keys in a given computation (as opposed to being required to materialize all asset
keys).
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.check_specs'>`property` check_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1119' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.check_specs" class="hash-link"></a></Link></dt>
<dd>
Returns the asset check specs defined on this AssetsDefinition, i.e. the checks that can
be executed while materializing the assets.
Return type: Iterable[AssetsCheckSpec]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.dependency_keys'>`property` dependency_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L914' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.dependency_keys" class="hash-link"></a></Link></dt>
<dd>
The asset keys which are upstream of any asset included in this
AssetsDefinition.
Type: Iterable[[AssetKey](#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.descriptions_by_key'>`property` descriptions_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L814' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.descriptions_by_key" class="hash-link"></a></Link></dt>
<dd>
Returns a mapping from the asset keys in this AssetsDefinition
to the descriptions assigned to them. If there is no assigned description for a given AssetKey,
it will not be present in this dictionary.
Type: Mapping[[AssetKey](#dagster.AssetKey), str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.group_names_by_key'>`property` group_names_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L805' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.group_names_by_key" class="hash-link"></a></Link></dt>
<dd>
Returns a mapping from the asset keys in this AssetsDefinition
to the group names assigned to them. If there is no assigned group name for a given AssetKey,
it will not be present in this dictionary.
Type: Mapping[[AssetKey](#dagster.AssetKey), str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.key'>`property` key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L867' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.key" class="hash-link"></a></Link></dt>
<dd>
The asset key associated with this AssetsDefinition. If this AssetsDefinition
has more than one asset key, this will produce an error.
Type: [AssetKey](#dagster.AssetKey)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.keys'>`property` keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L897' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.keys" class="hash-link"></a></Link></dt>
<dd>
The asset keys associated with this AssetsDefinition.
Type: AbstractSet[[AssetKey](#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.node_def'>`property` node_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L840' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.node_def" class="hash-link"></a></Link></dt>
<dd>
Returns the OpDefinition or GraphDefinition that is used to materialize
the assets in this AssetsDefinition.
Type: NodeDefinition
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.op'>`property` op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L827' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.op" class="hash-link"></a></Link></dt>
<dd>
Returns the OpDefinition that is used to materialize the assets in this
AssetsDefinition.
Type: [OpDefinition](ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.partitions_def'>`property` partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1066' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.partitions_def" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition for this AssetsDefinition (if any).
Type: Optional[[PartitionsDefinition](partitions.mdx#dagster.PartitionsDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L1568' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
The set of keys for resources that must be provided to this AssetsDefinition.
Type: Set[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetsDefinition.resource_defs'>`property` resource_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/assets_definition.py#L881' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetsDefinition.resource_defs" class="hash-link"></a></Link></dt>
<dd>
A mapping from resource name to ResourceDefinition for
the resources bound to this AssetsDefinition.
Type: Mapping[str, [ResourceDefinition](resources.mdx#dagster.ResourceDefinition)]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetKey'>`class` dagster.AssetKey <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_key.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetKey" class="hash-link"></a></Link></dt>
<dd>
Object representing the structure of an asset key. Takes in a sanitized string, list of
strings, or tuple of strings.
Example usage:
```python
from dagster import AssetKey
AssetKey("asset1")
AssetKey(["asset1"]) # same as the above
AssetKey(["prefix", "asset1"])
AssetKey(["prefix", "subprefix", "asset1"])
```
Parameters: <strong>path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – String, list of strings, or tuple of strings. A list of
strings represent the hierarchical structure of the asset_key.
<dl>
<dt><Link class="anchor" id='dagster.AssetKey.path'>`property` path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_key.py#L68' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetKey.path" class="hash-link"></a></Link></dt>
<dd>
The components of the asset key as a list of strings.
Type: Sequence[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.map_asset_specs'>dagster.map_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_spec.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.map_asset_specs" class="hash-link"></a></Link></dt>
<dd>
Map a function over a sequence of AssetSpecs or AssetsDefinitions, replacing specs in the sequence
or specs in an AssetsDefinitions with the result of the function.
Parameters:
- <strong>func</strong> (<em>Callable</em><em>[</em><em>[</em>[*AssetSpec*](#dagster.AssetSpec)<em>]</em><em>, </em>[*AssetSpec*](#dagster.AssetSpec)<em>]</em>) – The function to apply to each AssetSpec.
- <strong>iterable</strong> (<em>Iterable</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*AssetSpec*](#dagster.AssetSpec)<em>]</em><em>]</em>) – The sequence of AssetSpecs or AssetsDefinitions.
Returns:
A sequence of AssetSpecs or AssetsDefinitions with the function applied
to each spec.
Return type: Sequence[Union[[AssetsDefinition](#dagster.AssetsDefinition), [AssetSpec](#dagster.AssetSpec)]]
Examples:
```python
from dagster import AssetSpec, map_asset_specs
asset_specs = [
AssetSpec(key="my_asset"),
AssetSpec(key="my_asset_2"),
]
mapped_specs = map_asset_specs(lambda spec: spec.replace_attributes(owners=["nelson@hooli.com"]), asset_specs)
```
</dd>
</dl>
</div>
<div class="section" id="graph-backed-asset-definitions">
## Graph-backed asset definitions
Refer to the [Graph-backed asset](https://docs.dagster.io/guides/build/assets/defining-assets#graph-asset) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.graph_asset'>@dagster.graph_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py#L805' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.graph_asset" class="hash-link"></a></Link></dt>
<dd>
Creates a software-defined asset that’s computed using a graph of ops.
This decorator is meant to decorate a function that composes a set of ops or graphs to define
the dependencies between them.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the asset. If not provided, defaults to the name of the decorated function. The asset’s name must be a valid name in Dagster (ie only contains letters, numbers, and underscores) and may not contain Python reserved keywords.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the asset.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](#dagster.AssetIn)<em>]</em><em>]</em>) – A dictionary that maps input names to information about the input.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*ConfigMapping*](config.mdx#dagster.ConfigMapping)<em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) –
Describes how the graph underlying the asset is configured at runtime.
If a [`ConfigMapping`](config.mdx#dagster.ConfigMapping) object is provided, then the graph takes on the config schema of this object. The mapping will be applied at runtime to generate the config for the graph’s constituent nodes.
If a dictionary is provided, then it will be used as the default run config for the graph. This means it must conform to the config schema of the underlying nodes. Note that the values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the asset’s key is the concatenation of the key_prefix and the asset’s name, which defaults to the name of the decorated function. Each item in key_prefix must be a valid name in Dagster (ie only contains letters, numbers, and underscores) and may not contain Python reserved keywords.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
- <strong>hooks</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A set of hooks to attach to the asset. These hooks will be executed when the asset is materialized.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em>) – Dictionary of metadata to be associated with the asset.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing owners of the asset. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
- <strong>kinds</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing the kinds of the asset. These will be made visible in the Dagster UI.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – The AutomationCondition to use for this asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – The BackfillPolicy to use for this asset.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the code that generates this asset. In general, versions should be set only for code that deterministically produces the same output when given the same inputs.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>CoeercibleToAssetKey</em><em>]</em>) – The key for this asset. If provided, cannot specify key_prefix or name.
Examples:
```python
@op
def fetch_files_from_slack(context) -> pd.DataFrame:
...
@op
def store_files(files) -> None:
files.to_sql(name="slack_files", con=create_db_connection())
@graph_asset
def slack_files_table():
return store_files(fetch_files_from_slack())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.graph_multi_asset'>@dagster.graph_multi_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py#L1051' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.graph_multi_asset" class="hash-link"></a></Link></dt>
<dd>
Create a combined definition of multiple assets that are computed using the same graph of
ops, and the same upstream assets.
Each argument to the decorated function references an upstream asset that this asset depends on.
The name of the argument designates the name of the upstream asset.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the graph.
- <strong>outs</strong> – (Optional[Dict[str, AssetOut]]): The AssetOuts representing the produced assets.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](#dagster.AssetIn)<em>]</em><em>]</em>) – A dictionary that maps input names to information about the input.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the assets.
- <strong>hooks</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A list of hooks to attach to the asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – The backfill policy for the asset.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. This group name will be applied to all assets produced by this multi_asset.
- <strong>can_subset</strong> (<em>bool</em>) – Whether this asset’s computation can emit a subset of the asset keys based on the context.selected_assets argument. Defaults to False.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*ConfigMapping*](config.mdx#dagster.ConfigMapping)<em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) –
Describes how the graph underlying the asset is configured at runtime.
If a [`ConfigMapping`](config.mdx#dagster.ConfigMapping) object is provided, then the graph takes on the config schema of this object. The mapping will be applied at runtime to generate the config for the graph’s constituent nodes.
If a dictionary is provided, then it will be used as the default run config for the graph. This means it must conform to the config schema of the underlying nodes. Note that the values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
If no value is provided, then the config schema for the graph is the default (derived
</dd>
</dl>
</div>
<div class="section" id="multi-asset-definitions">
## Multi-asset definitions
Refer to the [Multi-asset](https://docs.dagster.io/guides/build/assets/defining-assets#multi-asset) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.multi_asset'>@dagster.multi_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/asset_decorator.py#L579' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multi_asset" class="hash-link"></a></Link></dt>
<dd>
Create a combined definition of multiple assets that are computed using the same op and same
upstream assets.
Each argument to the decorated function references an upstream asset that this asset depends on.
The name of the argument designates the name of the upstream asset.
You can set I/O managers keys, auto-materialize policies, freshness policies, group names, etc.
on an individual asset within the multi-asset by attaching them to the [`AssetOut`](#dagster.AssetOut)
corresponding to that asset in the <cite>outs</cite> parameter.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op.
- <strong>outs</strong> – (Optional[Dict[str, AssetOut]]): The AssetOuts representing the assets materialized by this function. AssetOuts detail the output, IO management, and core asset properties. This argument is required except when AssetSpecs are used.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](#dagster.AssetIn)<em>]</em><em>]</em>) – A dictionary that maps input names to information about the input.
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](#dagster.SourceAsset)<em>, </em>[*AssetKey*](#dagster.AssetKey)<em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – The assets that are upstream dependencies, but do not correspond to a parameter of the decorated function. If the AssetsDefinition for a multi_asset is provided, dependencies on all assets created by the multi_asset will be created.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The configuration schema for the asset’s underlying op. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource handles required by the underlying op.
- <strong>internal_asset_deps</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Set</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em><em>]</em>) – By default, it is assumed that all assets produced by a multi_asset depend on all assets that are consumed by that multi asset. If this default is not correct, you pass in a map of output names to a corrected set of AssetKeys that they depend on. Any AssetKeys in this list must be either used as input to the asset or produced within the op.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the assets.
- <strong>hooks</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A set of hooks to attach to the asset. These hooks will be executed when the asset is materialized.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – The backfill policy for the op that computes the asset.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>can_subset</strong> (<em>bool</em>) – If this asset’s computation can emit a subset of the asset keys based on the context.selected_asset_keys argument. Defaults to False.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A mapping of resource keys to resources. These resources will be initialized during execution, and can be accessed from the context within the body of the function.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. This group name will be applied to all assets produced by this multi_asset.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that computes the asset.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the code encapsulated by the multi-asset. If set, this is used as a default code version for all defined assets.
- <strong>specs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetSpec*](#dagster.AssetSpec)<em>]</em><em>]</em>) – The specifications for the assets materialized by this function.
- <strong>check_specs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckSpec*](asset-checks.mdx#dagster.AssetCheckSpec)<em>]</em><em>]</em>) – Specs for asset checks that execute in the decorated function after materializing the assets.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs this multi-asset’s execution.
- <strong>non_argument_deps</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Set</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>, </em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> Deprecated, use deps instead. Set of asset keys that are upstream dependencies, but do not pass an input to the multi_asset.
Examples:
```python
@multi_asset(
specs=[
AssetSpec("asset1", deps=["asset0"]),
AssetSpec("asset2", deps=["asset0"]),
]
)
def my_function():
asset0_value = load(path="asset0")
asset1_result, asset2_result = do_some_transformation(asset0_value)
write(asset1_result, path="asset1")
write(asset2_result, path="asset2")
# Or use IO managers to handle I/O:
@multi_asset(
outs={
"asset1": AssetOut(),
"asset2": AssetOut(),
}
)
def my_function(asset0):
asset1_value = do_some_transformation(asset0)
asset2_value = do_some_other_transformation(asset0)
return asset1_value, asset2_value
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.multi_observable_source_asset'>@dagster.multi_observable_source_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/source_asset_decorator.py#L212' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multi_observable_source_asset" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Defines a set of assets that can be observed together with the same function.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource handles required by the underlying op.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the assets.
- <strong>can_subset</strong> (<em>bool</em>) – If this asset’s computation can emit a subset of the asset keys based on the context.selected_assets argument. Defaults to False.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A mapping of resource keys to resources. These resources will be initialized during execution, and can be accessed from the context within the body of the function.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. This group name will be applied to all assets produced by this multi_asset.
- <strong>specs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetSpec*](#dagster.AssetSpec)<em>]</em><em>]</em>) – The specifications for the assets observed by this function.
- <strong>check_specs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckSpec*](asset-checks.mdx#dagster.AssetCheckSpec)<em>]</em><em>]</em>) – Specs for asset checks that execute in the decorated function after observing the assets.
Examples:
```python
@multi_observable_source_asset(
specs=[AssetSpec("asset1"), AssetSpec("asset2")],
)
def my_function():
yield ObserveResult(asset_key="asset1", metadata={"foo": "bar"})
yield ObserveResult(asset_key="asset2", metadata={"baz": "qux"})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetOut'>`class` dagster.AssetOut <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/job/asset_out.py#L38' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetOut" class="hash-link"></a></Link></dt>
<dd>
Defines one of the assets produced by a [`@multi_asset`](#dagster.multi_asset).
Parameters:
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the asset’s key is the concatenation of the key_prefix and the asset’s name. When using `@multi_asset`, the asset name defaults to the key of the “outs” dictionary Only one of the “key_prefix” and “key” arguments should be provided.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – The asset’s key. Only one of the “key_prefix” and “key” arguments should be provided.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Type</em><em>, </em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em><em>]</em><em>]</em>) – The type of this output. Should only be set if the correct type can not be inferred directly from the type signature of the decorated function.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the output.
- <strong>is_required</strong> (<em>bool</em>) – Whether the presence of this field is required. (default: True)
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The resource key of the IO manager used for this output. (default: “io_manager”).
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of the metadata for the output. For example, users can provide a file path if the data object will be stored in a filesystem, or provide information of a database table when it is going to load the data into the table.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The version of the code that generates this asset.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – AutomationCondition to apply to the specified asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to the specified asset.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing owners of the asset. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
- <strong>kinds</strong> (<em>Optional</em><em>[</em><em>set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of strings representing the kinds of the asset. These
will be made visible in the Dagster UI.
<dl>
<dt><Link class="anchor" id='dagster.AssetOut.from_spec'>`static` from_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/job/asset_out.py#L261' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetOut.from_spec" class="hash-link"></a></Link></dt>
<dd>
Builds an AssetOut from the passed spec.
Parameters:
- <strong>spec</strong> ([*AssetSpec*](#dagster.AssetSpec)) – The spec to build the AssetOut from.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Type</em><em>, </em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em><em>]</em>) – The type of this output. Should only be set if the correct type can not be inferred directly from the type signature of the decorated function.
- <strong>is_required</strong> (<em>bool</em>) – Whether the presence of this field is required. (default: True)
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The resource key of the IO manager used for this output. (default: “io_manager”).
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to the specified asset.
Returns: The AssetOut built from the spec.Return type: [AssetOut](#dagster.AssetOut)
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="source-assets">
## Source assets
Refer to the [External asset dependencies](https://docs.dagster.io/guides/build/assets/external-assets) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.SourceAsset'>`class` dagster.SourceAsset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/source_asset.py#L159' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SourceAsset" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.0.
Use AssetSpec instead. If using the SourceAsset io_manager_key property, use AssetSpec(...).with_io_manager_key(...)..
:::
A SourceAsset represents an asset that will be loaded by (but not updated by) Dagster.
Parameters:
- <strong>key</strong> (<em>Union</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em>) – The key of the asset.
- <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em>[*MetadataValue*](metadata.mdx#dagster.MetadataValue)<em>]</em>) – Metadata associated with the asset.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The key for the IOManager that will be used to load the contents of the asset when it’s used as an input to other assets inside a job.
- <strong>io_manager_def</strong> (<em>Optional</em><em>[</em>[*IOManagerDefinition*](io-managers.mdx#dagster.IOManagerDefinition)<em>]</em>) – <span className="flag flag-info">beta</span> (Beta) The definition of the IOManager that will be used to load the contents of the asset when it’s used as an input to other assets inside a job.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) resource definitions that may be required by the [`dagster.IOManagerDefinition`](io-managers.mdx#dagster.IOManagerDefinition) provided in the <cite>io_manager_def</cite> argument.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The description of the asset.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
- <strong>observe_fn</strong> (<em>Optional</em><em>[</em><em>SourceAssetObserveFunction</em><em>]</em>)
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>auto_observe_interval_minutes</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – While the asset daemon is turned on, a run of the observation function for this asset will be launched at this interval. <cite>observe_fn</cite> must be provided.
- <strong>freshness_policy</strong> ([*FreshnessPolicy*](#dagster.FreshnessPolicy)) – A constraint telling Dagster how often this asset is intended to be updated with respect to its root data.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
<dl>
<dt><Link class="anchor" id='dagster.SourceAsset.is_observable'>`property` is_observable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/source_asset.py#L361' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SourceAsset.is_observable" class="hash-link"></a></Link></dt>
<dd>
Whether the asset is observable.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SourceAsset.op'>`property` op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/source_asset.py#L334' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SourceAsset.op" class="hash-link"></a></Link></dt>
<dd>
The OpDefinition associated with the observation function of an observable
source asset.
Throws an error if the asset is not observable.
Type: [OpDefinition](ops.mdx#dagster.OpDefinition)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.observable_source_asset'>@dagster.observable_source_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/source_asset_decorator.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.observable_source_asset" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Create a <cite>SourceAsset</cite> with an associated observation function.
The observation function of a source asset is wrapped inside of an op and can be executed as
part of a job. Each execution generates an <cite>AssetObservation</cite> event associated with the source
asset. The source asset observation function should return a `DataVersion`,
a <cite>~dagster.DataVersionsByPartition</cite>, or an [`ObserveResult`](#dagster.ObserveResult).
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the source asset. If not provided, defaults to the name of the decorated function. The asset’s name must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the source asset’s key is the concatenation of the key_prefix and the asset’s name, which defaults to the name of the decorated function. Each item in key_prefix must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em>) – Metadata associated with the asset.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The key for the IOManager that will be used to load the contents of the source asset when it’s used as an input to other assets inside a job.
- <strong>io_manager_def</strong> (<em>Optional</em><em>[</em>[*IOManagerDefinition*](io-managers.mdx#dagster.IOManagerDefinition)<em>]</em>) – <span className="flag flag-info">beta</span> (Beta) The definition of the IOManager that will be used to load the contents of the source asset when it’s used as an input to other assets inside a job.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The description of the asset.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource keys required by the observe op.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) resource definitions that may be required by the [`dagster.IOManagerDefinition`](io-managers.mdx#dagster.IOManagerDefinition) provided in the <cite>io_manager_def</cite> argument.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for filtering and organizing. These tags are not attached to runs of the asset.
- <strong>observe_fn</strong> (<em>Optional</em><em>[</em><em>SourceAssetObserveFunction</em><em>]</em>) – Observation function for the source asset.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – A condition describing when Dagster should materialize this asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ObserveResult'>`class` dagster.ObserveResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/result.py#L109' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ObserveResult" class="hash-link"></a></Link></dt>
<dd>
An object representing a successful observation of an asset. These can be returned from an
@observable_source_asset decorated function to pass metadata.
Parameters:
- <strong>asset_key</strong> (<em>Optional</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – The asset key. Optional to include.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em>) – Metadata to record with the corresponding AssetObservation event.
- <strong>check_results</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckResult*](asset-checks.mdx#dagster.AssetCheckResult)<em>]</em><em>]</em>) – Check results to record with the corresponding AssetObservation event.
- <strong>data_version</strong> (<em>Optional</em><em>[</em><em>DataVersion</em><em>]</em>) – The data version of the asset that was observed.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags to record with the corresponding AssetObservation event.
</dd>
</dl>
</div>
<div class="section" id="dependencies">
## Dependencies
<dl>
<dt><Link class="anchor" id='dagster.AssetDep'>`class` dagster.AssetDep <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/definition/asset_dep.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetDep" class="hash-link"></a></Link></dt>
<dd>
Specifies a dependency on an upstream asset.
Parameters:
- <strong>asset</strong> (<em>Union</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>, </em><em>str</em><em>, </em>[*AssetSpec*](#dagster.AssetSpec)<em>, </em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](#dagster.SourceAsset)<em>]</em>) – The upstream asset to depend on.
- <strong>partition_mapping</strong> (<em>Optional</em><em>[</em>[*PartitionMapping*](partitions.mdx#dagster.PartitionMapping)<em>]</em>) – Defines what partitions to depend on in the upstream asset. If not provided and the upstream asset is partitioned, defaults to the default partition mapping for the partitions definition, which is typically maps partition keys to the same partition keys in upstream assets.
Examples:
```python
upstream_asset = AssetSpec("upstream_asset")
downstream_asset = AssetSpec(
"downstream_asset",
deps=[
AssetDep(
upstream_asset,
partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)
)
]
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetIn'>`class` dagster.AssetIn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/assets/job/asset_in.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetIn" class="hash-link"></a></Link></dt>
<dd>
Defines an asset dependency.
Parameters:
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the asset’s key is the concatenation of the key_prefix and the input name. Only one of the “key_prefix” and “key” arguments should be provided.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>]</em>) – The asset’s key. Only one of the “key_prefix” and “key” arguments should be provided.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of the metadata for the input. For example, if you only need a subset of columns from an upstream table, you could include that in metadata and the IO manager that loads the upstream table could use the metadata to determine which columns to load.
- <strong>partition_mapping</strong> (<em>Optional</em><em>[</em>[*PartitionMapping*](partitions.mdx#dagster.PartitionMapping)<em>]</em>) – Defines what partitions to depend on in the upstream asset. If not provided, defaults to the default partition mapping for the partitions definition, which is typically maps partition keys to the same partition keys in upstream assets.
- <strong>dagster_type</strong> ([*DagsterType*](types.mdx#dagster.DagsterType)) – Allows specifying type validation functions that will be executed on the input of the decorated function before it runs.
</dd>
</dl>
</div>
<div class="section" id="asset-jobs">
## Asset jobs
[Asset jobs](https://docs.dagster.io/guides/build/jobs/asset-jobs) enable the automation of asset materializations. Dagster’s [asset selection syntax](https://docs.dagster.io/guides/build/assets/asset-selection-syntax) can be used to select assets and assign them to a job.
<dl>
<dt><Link class="anchor" id='dagster.define_asset_job'>dagster.define_asset_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/unresolved_asset_job_definition.py#L260' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.define_asset_job" class="hash-link"></a></Link></dt>
<dd>
Creates a definition of a job which will either materialize a selection of assets or observe
a selection of source assets. This will only be resolved to a JobDefinition once placed in a
project.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name for the job.
- <strong>selection</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](#dagster.AssetSelection)<em>]</em>) –
The assets that will be materialized or observed when the job is run.
The selected assets must all be included in the assets that are passed to the assets argument of the Definitions object that this job is included on.
The string “my_asset*” selects my_asset and all downstream assets within the code location. A list of strings represents the union of all assets selected by strings within the list.
- <strong>config</strong> –
Describes how the Job is parameterized at runtime.
If no value is provided, then the schema for the job’s run config is a standard format based on its ops and resources.
If a dictionary is provided, then it must conform to the standard config schema, and it will be used as the job’s run config for the job whenever the job is executed. The values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the job and can be used for searching and filtering in the UI. Values that are not already strings will be serialized as JSON. If <cite>run_tags</cite> is not set, then the content of <cite>tags</cite> will also be automatically appended to the tags of any runs of this job.
- <strong>run_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that will be automatically attached to runs launched by this job. Values that are not already strings will be serialized as JSON. These tag values may be overwritten by tag values provided at invocation time. If <cite>run_tags</cite> is set, then <cite>tags</cite> are not automatically appended to the tags of any runs of this job.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the job. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A description for the Job.
- <strong>executor_def</strong> (<em>Optional</em><em>[</em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em>) – How this Job will be executed. Defaults to [`multi_or_in_process_executor`](execution.mdx#dagster.multi_or_in_process_executor), which can be switched between multi-process and in-process modes of execution. The default mode of execution is multi-process.
- <strong>hooks</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A set of hooks to be attached to each asset in the job. These hooks define logic that runs in response to events such as success or failure during the execution of individual assets.
- <strong>op_retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The default retry policy for all ops that compute assets in this job. Only used if retry policy is not defined on the asset definition.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – <span className="flag flag-warning">deprecated</span> (Deprecated) Defines the set of partitions for this job. Deprecated because partitioning is inferred from the selected assets, so setting this is redundant.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of strings representing owners of the job. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>[</em><em>AssetJobKey</em><em>]</em><em>]</em>) – (Preview) A job-scoped automation condition. When the condition becomes true, the job is launched by the automation condition sensor/daemon.
Returns: The job, which can be placed inside a project.Return type: UnresolvedAssetJobDefinition
Examples:
```python
# A job that targets all assets in the project:
@asset
def asset1():
...
Definitions(
assets=[asset1],
jobs=[define_asset_job("all_assets")],
)
# A job that targets a single asset
@asset
def asset1():
...
Definitions(
assets=[asset1],
jobs=[define_asset_job("all_assets", selection=[asset1])],
)
# A job that targets all the assets in a group:
Definitions(
assets=assets,
jobs=[define_asset_job("marketing_job", selection=AssetSelection.groups("marketing"))],
)
@observable_source_asset
def source_asset():
...
# A job that observes a source asset:
Definitions(
assets=assets,
jobs=[define_asset_job("observation_job", selection=[source_asset])],
)
# Resources are supplied to the assets, not the job:
@asset(required_resource_keys={"slack_client"})
def asset1():
...
Definitions(
assets=[asset1],
jobs=[define_asset_job("all_assets")],
resources={"slack_client": prod_slack_client},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection'>`class` dagster.AssetSelection <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection" class="hash-link"></a></Link></dt>
<dd>
An AssetSelection defines a query over a set of assets and asset checks, normally all that are defined in a project.
You can use the “|”, “&”, and “-” operators to create unions, intersections, and differences of selections, respectively.
AssetSelections are typically used with [`define_asset_job()`](#dagster.define_asset_job).
By default, selecting assets will also select all of the asset checks that target those assets.
Examples:
```python
# Select all assets in group "marketing":
AssetSelection.groups("marketing")
# Select all assets in group "marketing", as well as the asset with key "promotion":
AssetSelection.groups("marketing") | AssetSelection.assets("promotion")
# Select all assets in group "marketing" that are downstream of asset "leads":
AssetSelection.groups("marketing") & AssetSelection.assets("leads").downstream()
# Select a list of assets:
AssetSelection.assets(*my_assets_list)
# Select all assets except for those in group "marketing"
AssetSelection.all() - AssetSelection.groups("marketing")
# Select all assets which are materialized by the same op as "projections":
AssetSelection.assets("projections").required_multi_asset_neighbors()
# Select all assets in group "marketing" and exclude their asset checks:
AssetSelection.groups("marketing") - AssetSelection.all_asset_checks()
# Select all asset checks that target a list of assets:
AssetSelection.checks_for_assets(*my_assets_list)
# Select a specific asset check:
AssetSelection.checks(my_asset_check)
```
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.all'>`static` all <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L108' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.all" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all assets and their asset checks.
Parameters: <strong>include_sources</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> If True, then include all external assets.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.all_asset_checks'>`static` all_asset_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L119' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.all_asset_checks" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all asset checks.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.assets'>`static` assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L125' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.assets" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all of the provided assets and asset checks that target
them.
Parameters: <strong>*assets_defs</strong> (<em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – The assets to
select.
Examples:
```python
AssetSelection.assets(AssetKey(["a"]))
AssetSelection.assets("a")
AssetSelection.assets(AssetKey(["a"]), AssetKey(["b"]))
AssetSelection.assets("a", "b")
@asset
def asset1():
...
AssetSelection.assets(asset1)
asset_key_list = [AssetKey(["a"]), AssetKey(["b"])]
AssetSelection.assets(*asset_key_list)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.checks'>`static` checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L330' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.checks" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all of the provided asset checks or check keys.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.checks_for_assets'>`static` checks_for_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L315' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.checks_for_assets" class="hash-link"></a></Link></dt>
<dd>
Returns a selection with the asset checks that target the provided assets.
Parameters: <strong>*assets_defs</strong> (<em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – The assets to
select checks for.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.groups'>`static` groups <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L243' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.groups" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes materializable assets that belong to any of the
provided groups and all the asset checks that target them.
Parameters: <strong>include_sources</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> If True, then include external assets matching the group in the
selection.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.key_prefixes'>`static` key_prefixes <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L191' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.key_prefixes" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes assets that match any of the provided key prefixes and all the asset checks that target them.
Parameters: <strong>include_sources</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> If True, then include external assets matching the key prefix(es)
in the selection.
Examples:
```python
# match any asset key where the first segment is equal to "a" or "b"
# e.g. AssetKey(["a", "b", "c"]) would match, but AssetKey(["abc"]) would not.
AssetSelection.key_prefixes("a", "b")
# match any asset key where the first two segments are ["a", "b"] or ["a", "c"]
AssetSelection.key_prefixes(["a", "b"], ["a", "c"])
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.keys'>`static` keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L159' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.keys" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use AssetSelection.assets instead..
:::
Returns a selection that includes assets with any of the provided keys and all asset
checks that target them.
Deprecated: use AssetSelection.assets instead.
Examples:
```python
AssetSelection.keys(AssetKey(["a"]))
AssetSelection.keys("a")
AssetSelection.keys(AssetKey(["a"]), AssetKey(["b"]))
AssetSelection.keys("a", "b")
asset_key_list = [AssetKey(["a"]), AssetKey(["b"])]
AssetSelection.keys(*asset_key_list)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.tag'>`static` tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L259' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.tag" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes materializable assets that have the provided tag, and
all the asset checks that target them.
Parameters: <strong>include_sources</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> If True, then include external assets matching the group in the
selection.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.downstream'>downstream <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L345' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.downstream" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all assets that are downstream of any of the assets in
this selection, selecting the assets in this selection by default. Includes the asset checks targeting the returned assets. Iterates through each
asset in this selection and returns the union of all downstream assets.
depth (Optional[int]): If provided, then only include assets to the given depth. A depth
of 2 means all assets that are children or grandchildren of the assets in this
selection.
include_self (bool): If True, then include the assets in this selection in the result.
If the include_self flag is False, return each downstream asset that is not part of the
original selection. By default, set to True.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.materializable'>materializable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L420' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.materializable" class="hash-link"></a></Link></dt>
<dd>
Given an asset selection, returns a new asset selection that contains all of the assets
that are materializable. Removes any assets which are not materializable.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.required_multi_asset_neighbors'>required_multi_asset_neighbors <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L398' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.required_multi_asset_neighbors" class="hash-link"></a></Link></dt>
<dd>
Given an asset selection in which some assets are output from a multi-asset compute op
which cannot be subset, returns a new asset selection that contains all of the assets
required to execute the original asset selection. Includes the asset checks targeting the returned assets.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.roots'>roots <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L406' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.roots" class="hash-link"></a></Link></dt>
<dd>
Given an asset selection, returns a new asset selection that contains all of the root
assets within the original asset selection. Includes the asset checks targeting the returned assets.
A root asset is an asset that has no upstream dependencies within the asset selection.
The root asset can have downstream dependencies outside of the asset selection.
Because mixed selections of external and materializable assets are currently not supported,
keys corresponding to external assets will not be included as roots. To select external assets,
use the <cite>upstream_source_assets</cite> method.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.sinks'>sinks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L388' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.sinks" class="hash-link"></a></Link></dt>
<dd>
Given an asset selection, returns a new asset selection that contains all of the sink
assets within the original asset selection. Includes the asset checks targeting the returned assets.
A sink asset is an asset that has no downstream dependencies within the asset selection.
The sink asset can have downstream dependencies outside of the asset selection.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.sources'>sources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L427' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.sources" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use AssetSelection.roots instead..
:::
Given an asset selection, returns a new asset selection that contains all of the root
assets within the original asset selection. Includes the asset checks targeting the returned assets.
A root asset is a materializable asset that has no upstream dependencies within the asset
selection. The root asset can have downstream dependencies outside of the asset selection.
Because mixed selections of external and materializable assets are currently not supported,
keys corresponding to external assets will not be included as roots. To select external assets,
use the <cite>upstream_source_assets</cite> method.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.upstream'>upstream <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L364' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.upstream" class="hash-link"></a></Link></dt>
<dd>
Returns a selection that includes all materializable assets that are upstream of any of
the assets in this selection, selecting the assets in this selection by default. Includes
the asset checks targeting the returned assets. Iterates through each asset in this
selection and returns the union of all upstream assets.
Because mixed selections of external and materializable assets are currently not supported,
keys corresponding to external assets will not be included as upstream of regular assets.
Parameters:
- <strong>depth</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – If provided, then only include assets to the given depth. A depth of 2 means all assets that are parents or grandparents of the assets in this selection.
- <strong>include_self</strong> (<em>bool</em>) – If True, then include the assets in this selection in the result. If the include_self flag is False, return each upstream asset that is not part of the original selection. By default, set to True.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.upstream_source_assets'>upstream_source_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L442' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.upstream_source_assets" class="hash-link"></a></Link></dt>
<dd>
Given an asset selection, returns a new asset selection that contains all of the external
assets that are parents of assets in the original selection. Includes the asset checks
targeting the returned assets.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSelection.without_checks'>without_checks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_selection.py#L450' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSelection.without_checks" class="hash-link"></a></Link></dt>
<dd>
Removes all asset checks in the selection.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="code-locations">
## Code locations
Loading assets and asset jobs into a code location makes them available to Dagster tools like the UI, CLI, and GraphQL API.
A code location is automatically created when you [create a Dagster project](https://docs.dagster.io/guides/build/projects/creating-projects) with the [create-dagster CLI](https://docs.dagster.io/api/clis/create-dagster).
<dl>
<dt><Link class="anchor" id='dagster.load_assets_from_modules'>dagster.load_assets_from_modules <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_assets_from_modules.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_assets_from_modules" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of assets and source assets from the given modules.
Parameters:
- <strong>modules</strong> (<em>Iterable</em><em>[</em><em>ModuleType</em><em>]</em>) – The Python modules to look for assets inside.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Group name to apply to the loaded assets. The returned assets will be copies of the loaded objects, with the group name added.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Prefix to prepend to the keys of the loaded assets. The returned assets will be copies of the loaded objects, with the prefix prepended.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – AutomationCondition to apply to all the loaded assets.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to all the loaded assets.
- <strong>source_key_prefix</strong> (<em>bool</em>) – Prefix to prepend to the keys of loaded SourceAssets. The returned assets will be copies of the loaded objects, with the prefix prepended.
Returns: A list containing assets and source assets defined in the given modules.Return type: Sequence[Union[[AssetsDefinition](#dagster.AssetsDefinition), [SourceAsset](#dagster.SourceAsset)]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_assets_from_current_module'>dagster.load_assets_from_current_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_assets_from_modules.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_assets_from_current_module" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of assets, source assets, and cacheable assets from the module where
this function is called.
Parameters:
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Group name to apply to the loaded assets. The returned assets will be copies of the loaded objects, with the group name added.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Prefix to prepend to the keys of the loaded assets. The returned assets will be copies of the loaded objects, with the prefix prepended.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – AutomationCondition to apply to all the loaded assets.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to all the loaded assets.
- <strong>source_key_prefix</strong> (<em>bool</em>) – Prefix to prepend to the keys of loaded SourceAssets. The returned assets will be copies of the loaded objects, with the prefix prepended.
Returns: A list containing assets, source assets, and cacheable assets defined in the module.Return type: Sequence[Union[[AssetsDefinition](#dagster.AssetsDefinition), [SourceAsset](#dagster.SourceAsset), CachableAssetsDefinition]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_assets_from_package_module'>dagster.load_assets_from_package_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_assets_from_modules.py#L197' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_assets_from_package_module" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of assets and source assets that includes all asset
definitions, source assets, and cacheable assets in all sub-modules of the given package module.
A package module is the result of importing a package.
Parameters:
- <strong>package_module</strong> (<em>ModuleType</em>) – The package module to looks for assets inside.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Group name to apply to the loaded assets. The returned assets will be copies of the loaded objects, with the group name added.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Prefix to prepend to the keys of the loaded assets. The returned assets will be copies of the loaded objects, with the prefix prepended.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – AutomationCondition to apply to all the loaded assets.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to all the loaded assets.
- <strong>source_key_prefix</strong> (<em>bool</em>) – Prefix to prepend to the keys of loaded SourceAssets. The returned assets will be copies of the loaded objects, with the prefix prepended.
Returns: A list containing assets, source assets, and cacheable assets defined in the module.Return type: Sequence[Union[[AssetsDefinition](#dagster.AssetsDefinition), [SourceAsset](#dagster.SourceAsset), CacheableAssetsDefinition]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_assets_from_package_name'>dagster.load_assets_from_package_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_assets_from_modules.py#L256' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_assets_from_package_name" class="hash-link"></a></Link></dt>
<dd>
Constructs a list of assets, source assets, and cacheable assets that includes all asset
definitions and source assets in all sub-modules of the given package.
Parameters:
- <strong>package_name</strong> (<em>str</em>) – The name of a Python package to look for assets inside.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Group name to apply to the loaded assets. The returned assets will be copies of the loaded objects, with the group name added.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Prefix to prepend to the keys of the loaded assets. The returned assets will be copies of the loaded objects, with the prefix prepended.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – BackfillPolicy to apply to all the loaded assets.
- <strong>source_key_prefix</strong> (<em>bool</em>) – Prefix to prepend to the keys of loaded SourceAssets. The returned assets will be copies of the loaded objects, with the prefix prepended.
Returns: A list containing assets, source assets, and cacheable assets defined in the module.Return type: Sequence[Union[[AssetsDefinition](#dagster.AssetsDefinition), [SourceAsset](#dagster.SourceAsset), CacheableAssetsDefinition]]
</dd>
</dl>
</div>
<div class="section" id="observations">
## Observations
Refer to the [Asset observation](https://docs.dagster.io/guides/build/assets/metadata-and-tags/asset-observations) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.AssetObservation'>`class` dagster.AssetObservation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L373' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetObservation" class="hash-link"></a></Link></dt>
<dd>
Event that captures metadata about an asset at a point in time.
Parameters:
- <strong>asset_key</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](#dagster.AssetKey)<em>]</em>) – A key to identify the asset.
- <strong>partition</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of a partition of the asset that the metadata corresponds to.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A mapping containing tags for the observation.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>str</em><em>, </em><em>float</em><em>, </em><em>int</em><em>, </em>[*MetadataValue*](metadata.mdx#dagster.MetadataValue)<em>]</em><em>]</em><em>]</em>) – Arbitrary metadata about the asset. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
</dd>
</dl>
</div>
<div class="section" id="freshness-policies">
## Freshness policies
Freshness policies allow you to define freshness expectations for your assets and track their freshness state over time.
<dl>
<dt><Link class="anchor" id='dagster.FreshnessPolicy'>`class` dagster.FreshnessPolicy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/freshness.py#L52' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FreshnessPolicy" class="hash-link"></a></Link></dt>
<dd>
Base class for all freshness policies.
A freshness policy allows you to define expectations for the timing and frequency of asset materializations.
An asset with a defined freshness policy can take on different freshness states:
- `PASS`: The asset is passing its freshness policy.
- `WARN`: The asset is close to failing its freshness policy.
- `FAIL`: The asset is failing its freshness policy.
- `UNKNOWN`: The asset has no materialization events, and the freshness state cannot be determined.
If an asset does not have a freshness policy defined, it will have a freshness state of `NOT_APPLICABLE`.
This class provides static constructors for each of the supported freshness policy types. It is preferred to use these constructors to instantiate freshness policies, instead of instantiating the policy subtypes directly.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FreshnessPolicy.time_window'>`static` FreshnessPolicy.time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/freshness.py#L80' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FreshnessPolicy.time_window" class="hash-link"></a></Link></dt>
<dd>
Defines freshness with reference to a time window.
Parameters:
- <strong>fail_window</strong> – a timedelta that defines the failure window for the asset.
- <strong>warn_window</strong> – an optional timedelta that defines the warning window for the asset.
Returns: A `TimeWindowFreshnessPolicy` instance.
Examples:
```python
policy = FreshnessPolicy.time_window(
fail_window=timedelta(hours=24), warn_window=timedelta(hours=12)
)
```
This policy expects the asset to materialize at least once every 24 hours, and warns if the latest materialization is older than 12 hours.
- If it has been less than 12 hours since the latest materialization, the asset is passing its freshness policy, and will have a freshness state of `PASS`.
- If it has been between 12 and 24 hours since the latest materialization, the asset will have a freshness state of `WARN`.
- If it has been more than 24 hours since the latest materialization, the asset is failing its freshness policy, and will have a freshness state of `FAIL`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FreshnessPolicy.cron'>`static` FreshnessPolicy.cron <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/freshness.py#L109' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FreshnessPolicy.cron" class="hash-link"></a></Link></dt>
<dd>
Defines freshness with reference to a predetermined cron schedule.
Parameters:
- <strong>deadline_cron</strong> – a cron string that defines a deadline for the asset to be materialized.
- <strong>lower_bound_delta</strong> – a timedelta that defines the lower bound for when the asset could have been materialized. If a deadline cron tick has passed and the most recent materialization is older than (deadline cron tick timestamp - lower bound delta), the asset is considered stale until it materializes again.
- <strong>timezone</strong> – optionally provide a timezone for cron evaluation. IANA time zone strings are supported. If not provided, defaults to UTC.
Returns: A `CronFreshnessPolicy` instance.
Examples:
```python
policy = FreshnessPolicy.cron(
deadline_cron="0 10 * * *", # 10am daily
lower_bound_delta=timedelta(hours=1),
)
```
This policy expects the asset to materialize every day between 9:00 AM and 10:00 AM.
- If the asset is materialized at 9:30 AM, the asset is passing its freshness policy, and will have a freshness state of `PASS`. The asset will continue to pass the freshness policy until at least the deadline next day (10AM).
- If the asset is materialized at 9:59 AM, the asset is passing its freshness policy, and will have a freshness state of `PASS`. The asset will continue to pass the freshness policy until at least the deadline next day (10AM).
- If the asset is not materialized by 10:00 AM, the asset is failing its freshness policy, and will have a freshness state of `FAIL`. The asset will continue to fail the freshness policy until it is materialized again.
- If the asset is then materialized at 10:30AM, it will pass the freshness policy again until at least the deadline the next day (10AM).
Keep in mind that the policy will always look at the last completed cron tick.
So in the example above, if asset freshness is evaluated at 9:59 AM, the policy will still consider the previous day’s 9-10AM window.
</dd>
</dl>
</div>
<div class="section" id="declarative-automation">
## Declarative Automation
Refer to the [Declarative Automation](https://docs.dagster.io/guides/automate/declarative-automation) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition'>`class` dagster.AutomationCondition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L74' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition" class="hash-link"></a></Link></dt>
<dd>
An AutomationCondition represents a condition of an asset that impacts whether it should be
automatically executed. For example, you can have a condition which becomes true whenever the
code version of the asset is changed, or whenever an upstream dependency is updated.
```python
from dagster import AutomationCondition, asset
@asset(automation_condition=AutomationCondition.on_cron("0 0 * * *"))
def my_asset(): ...
```
AutomationConditions may be combined together into expressions using a variety of operators.
```python
from dagster import AssetSelection, AutomationCondition, asset
# any dependencies from the "important" group are missing
any_important_deps_missing = AutomationCondition.any_deps_match(
AutomationCondition.missing(),
).allow(AssetSelection.groups("important"))
# there is a new code version for this asset since the last time it was requested
new_code_version = AutomationCondition.code_version_changed().since(
AutomationCondition.newly_requested()
)
# there is a new code version and no important dependencies are missing
my_condition = new_code_version & ~any_important_deps_missing
@asset(automation_condition=my_condition)
def my_asset(): ...
```
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.all_checks_match'>`static` all_checks_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L486' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.all_checks_match" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for an asset partition if all of its checks
evaluate to True for the given condition.
Parameters:
- <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated against this asset’s checks.
- <strong>blocking_only</strong> (<em>bool</em>) – Determines if this condition will only be evaluated against blocking checks. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.all_deps_blocking_checks_passed'>`static` all_deps_blocking_checks_passed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L891' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.all_deps_blocking_checks_passed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for any partition where all upstream
blocking checks have passed, or will be requested on this tick.
In-tick requests are allowed to enable creating runs that target both a parent with
blocking checks and a child. Even though the checks have not currently passed, if
they fail within the run, the run machinery will prevent the child from being
materialized.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.all_deps_match'>`static` all_deps_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.all_deps_match" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for a if at least one partition
of the all of the target’s dependencies evaluate to True for the given condition.
Parameters: <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated against
this target’s dependencies.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.all_deps_updated_since_cron'>`static` all_deps_updated_since_cron <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L909' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.all_deps_updated_since_cron" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomatonCondition that is true if all of the target’s dependencies have
updated since the latest tick of the provided cron schedule.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.all_job_root_assets_match'>`static` all_job_root_assets_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L532' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.all_job_root_assets_match" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Returns an AutomationCondition that is true for a job if all of its root assets
evaluate to True for the given condition.
Only the job’s root assets (those with no in-job parents) are evaluated; downstream
assets are inferred via lookahead.
Parameters: <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated
against this job’s assets.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_checks_match'>`static` any_checks_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L465' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_checks_match" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for if at least one of the target’s
checks evaluate to True for the given condition.
Parameters:
- <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated against this asset’s checks.
- <strong>blocking_only</strong> (<em>bool</em>) – Determines if this condition will only be evaluated against blocking checks. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_deps_in_progress'>`static` any_deps_in_progress <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L881' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_deps_in_progress" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target has at least one dependency
that is in progress.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_deps_match'>`static` any_deps_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L433' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_deps_match" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for a if at least one partition
of the any of the target’s dependencies evaluate to True for the given condition.
Parameters: <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated against
this target’s dependencies.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_deps_missing'>`static` any_deps_missing <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L871' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_deps_missing" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target has at least one dependency
that is missing, and will not be requested on this tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_deps_updated'>`static` any_deps_updated <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L851' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_deps_updated" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target has at least one dependency
that has updated since the previous tick, or will be requested on this tick.
Will ignore parent updates if the run that updated the parent also plans to update
the asset or check that this condition is applied to.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_downstream_conditions'>`static` any_downstream_conditions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L986' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_downstream_conditions" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns an AutomationCondition which represents the union of all distinct downstream conditions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.any_job_root_assets_match'>`static` any_job_root_assets_match <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L507' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.any_job_root_assets_match" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Returns an AutomationCondition that is true for a job if at least one of its root
assets evaluates to True for the given condition.
Only the job’s root assets (those with no in-job parents) are evaluated; downstream
assets are inferred via lookahead.
Parameters: <strong>condition</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The AutomationCondition that will be evaluated
against this job’s assets.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.asset_matches'>`static` asset_matches <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L419' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.asset_matches" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if this condition is true for the given key.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.backfill_in_progress'>`static` backfill_in_progress <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L579' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.backfill_in_progress" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target is part of an in-progress backfill
that has not yet executed it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.check_failed'>`static` check_failed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L621' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.check_failed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for an asset check if it has evaluated against
the latest materialization of an asset and failed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.check_passed'>`static` check_passed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L611' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.check_passed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for an asset check if it has evaluated against
the latest materialization of an asset and passed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.code_version_changed'>`static` code_version_changed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L774' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.code_version_changed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target’s code version has been changed
since the previous tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.cron_tick_passed'>`static` cron_tick_passed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L798' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.cron_tick_passed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is whenever a cron tick of the provided schedule is passed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.data_version_changed'>`static` data_version_changed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L786' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.data_version_changed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target’s data version has been changed
since the previous tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.eager'>`static` eager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L924' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.eager" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition which will cause a target to be executed if any of
its dependencies update, and will execute missing partitions if they become missing
after this condition is applied to the target.
This will not execute targets that have any missing or in progress dependencies, or
are currently in progress.
For time partitioned assets, only the latest time partition will be considered.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.execution_failed'>`static` execution_failed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L591' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.execution_failed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the latest execution of the target failed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.freshness_failed'>`static` freshness_failed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L835' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.freshness_failed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target’s freshness is FAIL.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.freshness_passed'>`static` freshness_passed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L815' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.freshness_passed" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target’s freshness is PASS.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.freshness_warned'>`static` freshness_warned <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L825' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.freshness_warned" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target’s freshness is WARN.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.in_latest_time_window'>`static` in_latest_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L641' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.in_latest_time_window" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true when the target it is within the latest
time window.
Parameters: <strong>lookback_delta</strong> (<em>Optional</em><em>, </em><em>datetime.timedelta</em>) – If provided, the condition will
return all partitions within the provided delta of the end of the latest time window.
For example, if this is used on a daily-partitioned asset with a lookback_delta of
48 hours, this will return the latest two partitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.in_progress'>`static` in_progress <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L601' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.in_progress" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true for an asset partition if it is part of an
in-progress run or backfill that has not yet executed it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.initial_evaluation'>`static` initial_evaluation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L631' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.initial_evaluation" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true on the first evaluation of the expression.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.missing'>`static` missing <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L557' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.missing" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target has not been executed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.newly_missing'>`static` newly_missing <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L845' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.newly_missing" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true on the tick that the target becomes missing.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.newly_requested'>`static` newly_requested <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L764' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.newly_requested" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target was requested on the previous tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.newly_updated'>`static` newly_updated <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L671' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.newly_updated" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target has been updated since the previous tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.on_cron'>`static` on_cron <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L946' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.on_cron" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition which will cause a target to be executed on a given
cron schedule, after all of its dependencies have been updated since the latest
tick of that cron schedule.
For time partitioned assets, only the latest time partition will be considered.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.on_missing'>`static` on_missing <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L965' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.on_missing" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition which will execute partitions of the target that
are added after this condition is applied to the asset.
This will not execute targets that have any missing dependencies.
For time partitioned assets, only the latest time partition will be considered.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.run_in_progress'>`static` run_in_progress <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L567' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.run_in_progress" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target is part of an in-progress run
that has not yet executed it.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.will_be_requested'>`static` will_be_requested <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L661' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.will_be_requested" class="hash-link"></a></Link></dt>
<dd>
Returns an AutomationCondition that is true if the target will be requested this tick.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationCondition.replace'>replace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L404' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationCondition.replace" class="hash-link"></a></Link></dt>
<dd>
Replaces all instances of `old` across any sub-conditions with `new`.
If `old` is a string, then conditions with a label or name matching
that string will be replaced.
Parameters:
- <strong>old</strong> (<em>Union</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>, </em><em>str</em><em>]</em>) – The condition to replace.
- <strong>new</strong> ([*AutomationCondition*](#dagster.AutomationCondition)) – The condition to replace with.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationResult'>`class` dagster.AutomationResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition.py#L1060' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationResult" class="hash-link"></a></Link></dt>
<dd>
The result of evaluating an AutomationCondition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AutomationConditionSensorDefinition'>`class` dagster.AutomationConditionSensorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L826' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AutomationConditionSensorDefinition" class="hash-link"></a></Link></dt>
<dd>
Targets a set of assets and repeatedly evaluates all the AutomationConditions on all of
those assets to determine which to request runs for.
Parameters:
- <strong>name</strong> – The name of the sensor.
- <strong>target</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](#dagster.AssetSelection)<em>]</em>) – A selection of assets to evaluate AutomationConditions of and request runs for.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>run_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Tags that will be automatically attached to runs launched by this sensor.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The frequency at which to try to evaluate the sensor. The actual interval will be longer if the sensor evaluation takes longer than the provided interval.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>emit_backfills</strong> (<em>bool</em>) – If set to True, will emit a backfill on any tick where more than one partition of any single asset is requested, rather than individual runs. Defaults to True.
- <strong>use_user_code_server</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> (Beta) If set to True, this sensor will be evaluated in the user code server, rather than the AssetDaemon. This enables evaluating custom AutomationCondition subclasses, and ensures that the condition definitions will remain in sync with your user code version, eliminating version skew. Note: currently a maximum of 500 assets or checks may be targeted at a time by a sensor that has this value set.
- <strong>default_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](#dagster.AutomationCondition)<em>]</em>) – <span className="flag flag-info">beta</span> (Beta) If provided, this condition will be used for any selected assets or asset checks which do not have an automation condition defined. Requires <cite>use_user_code_server</cite> to be set to <cite>True</cite>.
Examples:
```python
import dagster as dg
# automation condition sensor that defaults to running
defs1 = dg.Definitions(
assets=...,
sensors=[
dg.AutomationConditionSensorDefinition(
name="automation_condition_sensor",
target=dg.AssetSelection.all(),
default_status=dg.DefaultSensorStatus.RUNNING,
),
]
)
# one automation condition sensor per group
defs2 = dg.Definitions(
assets=...,
sensors=[
dg.AutomationConditionSensorDefinition(
name="raw_data_automation_condition_sensor",
target=dg.AssetSelection.groups("raw_data"),
),
dg.AutomationConditionSensorDefinition(
name="ml_automation_condition_sensor",
target=dg.AssetSelection.groups("machine_learning"),
),
]
)
```
</dd>
</dl>
<div class="section" id="testing-declarative-automations">
### Testing Declarative Automations
<dl>
<dt><Link class="anchor" id='dagster.evaluate_automation_conditions'>dagster.evaluate_automation_conditions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition_tester.py#L76' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.evaluate_automation_conditions" class="hash-link"></a></Link></dt>
<dd>
Evaluates the AutomationConditions of the provided assets, returning the results as an instance
of [`EvaluateAutomationConditionsResult`](#dagster.EvaluateAutomationConditionsResult). Intended for use in unit tests.
Parameters:
- <strong>defs</strong> (<em>Union</em><em>[</em>[*Definitions*](definitions.mdx#dagster.Definitions)<em>, </em><em>Sequence</em><em>[</em><em>AssetsDefinitions</em><em>]</em><em>]</em>) – The definitions to evaluate the conditions of.
- <strong>instance</strong> ([*DagsterInstance*](internals.mdx#dagster.DagsterInstance)) – The instance to evaluate against.
- <strong>asset_selection</strong> ([*AssetSelection*](#dagster.AssetSelection)) – The selection of assets within defs to evaluate against. Defaults to all assets.
- <strong>evaluation_time</strong> (<em>Optional</em><em>[</em><em>datetime.datetime</em><em>]</em>) – The time to use for the evaluation. Defaults to the true current time.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>AssetDaemonCursor</em><em>]</em>) – The cursor for the computation. If you are evaluating multiple ticks within a test, this value should be supplied from the <cite>cursor</cite> property of the returned <cite>result</cite> object.
Examples:
<strong>Missing asset</strong> — an asset with `eager()` that has never been materialized is
requested on the first tick, then not re-requested on the second:
```python
import dagster as dg
@dg.asset(automation_condition=dg.AutomationCondition.eager())
def my_asset(): ...
def test_missing_asset():
instance = dg.DagsterInstance.ephemeral()
result = dg.evaluate_automation_conditions(defs=[my_asset], instance=instance)
assert result.total_requested == 1
result = dg.evaluate_automation_conditions(
defs=[my_asset], instance=instance, cursor=result.cursor
)
assert result.total_requested == 0
```
<strong>Same code location</strong> — upstream and downstream in the same `Definitions`;
materialize the upstream between ticks:
```python
import dagster as dg
@dg.asset
def upstream(): ...
@dg.asset(deps=[upstream], automation_condition=dg.AutomationCondition.eager())
def downstream(): ...
def test_eager_condition():
instance = dg.DagsterInstance.ephemeral()
result = dg.evaluate_automation_conditions(
defs=[upstream, downstream], instance=instance
)
assert result.total_requested == 0
dg.materialize([upstream], instance=instance)
result = dg.evaluate_automation_conditions(
defs=[upstream, downstream], instance=instance, cursor=result.cursor
)
assert result.total_requested == 1
```
<strong>Cross-code-location</strong> — represent an external upstream as an `AssetSpec` and
record its materialization with `report_runless_asset_event`:
```python
import dagster as dg
external_upstream = dg.AssetSpec(key="external_upstream")
@dg.asset(
deps=[external_upstream],
automation_condition=dg.AutomationCondition.eager(),
)
def my_asset(): ...
def test_cross_location_condition():
instance = dg.DagsterInstance.ephemeral()
result = dg.evaluate_automation_conditions(
defs=[external_upstream, my_asset], instance=instance
)
assert result.total_requested == 0
instance.report_runless_asset_event(
dg.AssetMaterialization(asset_key="external_upstream")
)
result = dg.evaluate_automation_conditions(
defs=[external_upstream, my_asset], instance=instance, cursor=result.cursor
)
assert result.total_requested == 1
result = dg.evaluate_automation_conditions(
defs=[external_upstream, my_asset], instance=instance, cursor=result.cursor
)
assert result.total_requested == 0
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EvaluateAutomationConditionsResult'>`class` dagster.EvaluateAutomationConditionsResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition_tester.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EvaluateAutomationConditionsResult" class="hash-link"></a></Link></dt>
<dd>
Returned by [`evaluate_automation_conditions()`](#dagster.evaluate_automation_conditions).
<dl>
<dt><Link class="anchor" id='dagster.EvaluateAutomationConditionsResult.get_num_requested'>get_num_requested <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition_tester.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EvaluateAutomationConditionsResult.get_num_requested" class="hash-link"></a></Link></dt>
<dd>
Returns the number of asset partitions requested for the given asset during this evaluation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EvaluateAutomationConditionsResult.get_requested_partitions'>get_requested_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition_tester.py#L65' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EvaluateAutomationConditionsResult.get_requested_partitions" class="hash-link"></a></Link></dt>
<dd>
Returns the specific partition keys requested for the given asset during this evaluation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EvaluateAutomationConditionsResult.total_requested'>`property` total_requested <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/declarative_automation/automation_condition_tester.py#L58' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EvaluateAutomationConditionsResult.total_requested" class="hash-link"></a></Link></dt>
<dd>
Returns the total number of asset partitions requested during this evaluation.
</dd>
</dl>
</dd>
</dl>
</div></div>
<div class="section" id="asset-values">
## Asset values
<dl>
<dt><Link class="anchor" id='dagster.AssetValueLoader'>`class` dagster.AssetValueLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/asset_value_loader.py#L29' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetValueLoader" class="hash-link"></a></Link></dt>
<dd>
Caches resource definitions that are used to load asset values across multiple load
invocations.
Should not be instantiated directly. Instead, use
[`get_asset_value_loader()`](repositories.mdx#dagster.RepositoryDefinition.get_asset_value_loader).
<dl>
<dt><Link class="anchor" id='dagster.AssetValueLoader.load_asset_value'>load_asset_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/asset_value_loader.py#L72' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetValueLoader.load_asset_value" class="hash-link"></a></Link></dt>
<dd>
Loads the contents of an asset as a Python object.
Invokes <cite>load_input</cite> on the [`IOManager`](io-managers.mdx#dagster.IOManager) associated with the asset.
Parameters:
- <strong>asset_key</strong> (<em>Union</em><em>[</em>[*AssetKey*](#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em>) – The key of the asset to load.
- <strong>python_type</strong> (<em>Optional</em><em>[</em><em>Type</em><em>]</em>) – The python type to load the asset as. This is what will be returned inside <cite>load_input</cite> by <cite>context.dagster_type.typing_type</cite>.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The partition of the asset to load.
- <strong>partition_key_range</strong> (<em>Optional</em><em>[</em>[*PartitionKeyRange*](partitions.mdx#dagster.PartitionKeyRange)<em>]</em>) – A range of partition keys to load. Mutually exclusive with `partition_key`.
- <strong>input_definition_metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Input metadata to pass to the [`IOManager`](io-managers.mdx#dagster.IOManager) (is equivalent to setting the metadata argument in <cite>In</cite> or <cite>AssetIn</cite>).
- <strong>resource_config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – A dictionary of resource configurations to be passed to the [`IOManager`](io-managers.mdx#dagster.IOManager).
Returns: The contents of an asset as a Python object.
</dd>
</dl>
</dd>
</dl>
</div></div>
---
---
title: 'components'
sidebar_position: 1000
title_meta: 'components API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'components Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="components">
# Components
<div class="section" id="building-custom-components">
## Building custom Components
<dl>
<dt><Link class="anchor" id='dagster.Component'>`class` dagster.Component <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/component/component.py#L95' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Component" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for creating Dagster components.
Components are the primary building blocks for programmatically creating Dagster
definitions. They enable building multiple interrelated definitions for specific use cases,
provide schema-based configuration, and built-in scaffolding support to simplify
component instantiation in projects. Components are automatically discovered by
Dagster tooling and can be instantiated from YAML configuration files or Python code that
conform to the declared schema.
Key Capabilities:
- <strong>Definition Factory</strong>: Creates Dagster assets, jobs, schedules, and other definitions
- <strong>Schema-Based Configuration</strong>: Optional parameterization via YAML or Python objects
- <strong>Scaffolding Support</strong>: Custom project structure generation via `dg scaffold` commands
- <strong>Tool Integration</strong>: Automatic discovery by Dagster CLI and UI tools
- <strong>Testing Utilities</strong>: Built-in methods for testing component behavior
Implementing a component:
- Every component must implement the `build_defs()` method, which serves as a factory for creating Dagster definitions.
- Components can optionally inherit from `Resolvable` to add schema-based configuration capabilities, enabling parameterization through YAML files or structured Python objects.
- Components can attach a custom scaffolder with the `@scaffold_with` decorator.
Examples:
Simple component with hardcoded definitions:
```python
import dagster as dg
class SimpleDataComponent(dg.Component):
"""Component that creates a toy, hardcoded data processing asset."""
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
@dg.asset
def raw_data():
return [1, 2, 3, 4, 5]
@dg.asset
def processed_data(raw_data):
return [x * 2 for x in raw_data]
return dg.Definitions(assets=[raw_data, processed_data])
```
Configurable component with schema:
```python
import dagster as dg
from typing import List
class DatabaseTableComponent(dg.Component, dg.Resolvable, dg.Model):
"""Component for creating assets from database tables."""
table_name: str
columns: List[str]
database_url: str = "postgresql://localhost/mydb"
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
@dg.asset(key=f"{self.table_name}_data")
def table_asset():
# Use self.table_name, self.columns, etc.
return execute_query(f"SELECT {', '.join(self.columns)} FROM {self.table_name}")
return dg.Definitions(assets=[table_asset])
```
Using the component in a YAML file (`defs.yaml`):
```yaml
type: my_project.components.DatabaseTableComponent
attributes:
table_name: "users"
columns: ["id", "name", "email"]
database_url: "postgresql://prod-db/analytics"
```
Component Discovery:
Components are automatically discovered by Dagster tooling when defined in modules
specified in your project’s `pyproject.toml` registry configuration:
```toml
[tool.dagster]
module_name = "my_project"
registry_modules = ["my_project.components"]
```
This enables CLI commands like:
```bash
dg list components # List all available components in the Python environment
dg scaffold defs MyComponent path/to/component # Generate component instance with scaffolding
```
Schema and Configuration:
To make a component configurable, inherit from both `Component` and `Resolvable`,
along with a model base class. Pydantic models and dataclasses are supported largely
so that pre-existing code can be used as schema without having to modify it. We recommend
using `dg.Model` for new components, which wraps Pydantic with Dagster defaults for better
developer experience.
- `dg.Model`: Recommended for new components (wraps Pydantic with Dagster defaults)
- `pydantic.BaseModel`: Direct Pydantic usage
- `@dataclass`: Python dataclasses with validation
Custom Scaffolding:
Components can provide custom scaffolding behavior using the `@scaffold_with` decorator:
```python
import textwrap
import dagster as dg
from dagster.components import Scaffolder, ScaffoldRequest
class DatabaseComponentScaffolder(Scaffolder):
def scaffold(self, request: ScaffoldRequest) -> None:
# Create component directory
component_dir = request.target_path
component_dir.mkdir(parents=True, exist_ok=True)
# Generate defs.yaml with template
defs_file = component_dir / "defs.yaml"
defs_file.write_text(
textwrap.dedent(
f'''
type: {request.type_name}
attributes:
table_name: "example_table"
columns: ["id", "name"]
database_url: "${{DATABASE_URL}}"
'''.strip()
)
)
# Generate SQL query template
sql_file = component_dir / "query.sql"
sql_file.write_text("SELECT * FROM example_table;")
@dg.scaffold_with(DatabaseComponentScaffolder)
class DatabaseTableComponent(dg.Component, dg.Resolvable, dg.Model):
table_name: str
columns: list[str]
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
# Component implementation
pass
```
See also: - [`dagster.Definitions`](definitions.mdx#dagster.Definitions)
- [`dagster.ComponentLoadContext`](#dagster.ComponentLoadContext)
- [`dagster.components.resolved.base.Resolvable`](#dagster.Resolvable)
- [`dagster.Model`](#dagster.Model)
- `dagster.scaffold_with()`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StateBackedComponent'>`class` dagster.StateBackedComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/component/state_backed_component.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StateBackedComponent" class="hash-link"></a></Link></dt>
<dd>
Base class for components that depend on external state that needs to be fetched and cached.
State-backed components are designed for integrations where Dagster definitions depend on
information from external systems (like APIs or compiled artifacts) rather than just code
and configuration files. The component framework manages the lifecycle of fetching, storing,
and loading this state.
Subclasses must implement:
- `write_state_to_path`: Fetches state from external sources and writes it to a local path
- `build_defs_from_state`: Builds Dagster definitions from the cached state
- `defs_state_config`: Property that returns configuration for state management
Example:
```python
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
import dagster as dg
from dagster.components import DefsStateConfig, DefsStateConfigArgs, ResolvedDefsStateConfig
@dataclass
class MyStateBackedComponent(dg.StateBackedComponent):
base_url: str
defs_state: ResolvedDefsStateConfig = DefsStateConfigArgs.local_filesystem()
@property
def defs_state_config(self) -> DefsStateConfig:
return DefsStateConfig.from_args(
self.defs_state, default_key=f"MyComponent[{self.base_url}]"
)
def write_state_to_path(self, state_path: Path) -> None:
# Fetch table metadata from external API
response = requests.get(f"{self.base_url}/api/tables")
tables = response.json()
# Write state to file as JSON
state_path.write_text(json.dumps(tables))
def build_defs_from_state(
self, context: dg.ComponentLoadContext, state_path: Optional[Path]
) -> dg.Definitions:
if state_path is None:
return dg.Definitions()
# Read cached state
tables = json.loads(state_path.read_text())
# Create one asset per table found in the state
assets = []
for table in tables:
@dg.asset(key=dg.AssetKey(table["name"]))
def table_asset():
# Fetch and return the actual table data
return fetch_table_data(table["name"])
assets.append(table_asset)
return dg.Definitions(assets=assets)
```
YAML configuration:
```yaml
# defs.yaml
type: my_package.MyStateBackedComponent
attributes:
base_url: "{{ env.MY_API_URL }}"
defs_state:
management_type: LOCAL_FILESYSTEM
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Resolvable'>`class` dagster.Resolvable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/resolved/base.py#L49' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Resolvable" class="hash-link"></a></Link></dt>
<dd>
Base class for making a class resolvable from yaml.
This framework is designed to allow complex nested objects to be resolved
from yaml documents. This allows for a single class to be instantiated from
either yaml or python without limiting the types of fields that can exist on
the python class.
Key Features:
- <strong>Automatic yaml schema derivation</strong>: A pydantic model is automatically generated from the class definition using its fields or <cite>__init__</cite> arguments and their annotations.
- <strong>Jinja template resolution</strong>: Fields in the yaml document may be templated strings, which are rendered from the available scope and may be arbitrary python objects.
- <strong>Customizable resolution behavior</strong>: Each field can customize how it is resolved from the yaml document using a <cite>:py:class:~dagster.Resolver</cite>.
Resolvable subclasses must be one of the following:
* pydantic model
* @dataclass
* plain class with an annotated <cite>__init__</cite>
* @record
Example:
```python
import datetime
from typing import Annotated
import dagster as dg
def resolve_timestamp(
context: dg.ResolutionContext,
raw_timestamp: str,
) -> datetime.datetime:
return datetime.datetime.fromisoformat(
context.resolve_value(raw_timestamp, as_type=str),
)
# the yaml field will be a string, which is then parsed into a datetime object
ResolvedTimestamp = Annotated[
datetime.datetime,
dg.Resolver(resolve_timestamp, model_field_type=str),
]
class MyClass(dg.Resolvable, dg.Model):
event: str
start_timestamp: ResolvedTimestamp
end_timestamp: ResolvedTimestamp
# python instantiation
in_python = MyClass(
event="test",
start_timestamp=datetime.datetime(2021, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc),
end_timestamp=datetime.datetime(2021, 1, 2, 0, 0, 0, tzinfo=datetime.timezone.utc),
)
# yaml instantiation
in_yaml = MyClass.resolve_from_yaml(
'''
event: test
start_timestamp: '{{ start_year }}-01-01T00:00:00Z'
end_timestamp: '{{ end_timestamp }}'
''',
scope={
# string templating
"start_year": "2021",
# object templating
"end_timestamp": in_python.end_timestamp,
},
)
assert in_python == in_yaml
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResolutionContext'>`class` dagster.ResolutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/resolved/context.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResolutionContext" class="hash-link"></a></Link></dt>
<dd>
The context available to Resolver functions when “resolving” from yaml in to a Resolvable object.
This class should not be instantiated directly.
Provides a <cite>resolve_value</cite> method that can be used to resolve templated values in a nested object before
being transformed into the final Resolvable object. This is typically invoked inside a
[`Resolver`](#dagster.Resolver)’s <cite>resolve_fn</cite> to ensure that jinja-templated values are turned into their
respective python types using the available template variables.
Example:
```python
import datetime
import dagster as dg
def resolve_timestamp(
context: dg.ResolutionContext,
raw_timestamp: str,
) -> datetime.datetime:
return datetime.datetime.fromisoformat(
context.resolve_value(raw_timestamp, as_type=str),
)
```
<dl>
<dt><Link class="anchor" id='dagster.ResolutionContext.resolve_value'>resolve_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/resolved/context.py#L188' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResolutionContext.resolve_value" class="hash-link"></a></Link></dt>
<dd>
Recursively resolves templated values in a nested object. This is typically
invoked inside a [`Resolver`](#dagster.Resolver)’s <cite>resolve_fn</cite> to resolve all
nested template values in the input object.
Parameters:
- <strong>val</strong> (<em>Any</em>) – The value to resolve.
- <strong>as_type</strong> (<em>Optional</em><em>[</em><em>type</em><em>]</em>) – If provided, the type to cast the resolved value to. Used purely for type hinting and does not impact runtime behavior.
Returns: The input value after all nested template values have been resolved.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Resolver'>`class` dagster.Resolver <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/resolved/model.py#L88' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Resolver" class="hash-link"></a></Link></dt>
<dd>
Contains information on how to resolve a value from YAML into the corresponding `Resolved` class field.
You can attach a resolver to a field’s type annotation to control how the value is resolved.
Example:
```python
import datetime
from typing import Annotated
import dagster as dg
def resolve_timestamp(
context: dg.ResolutionContext,
raw_timestamp: str,
) -> datetime.datetime:
return datetime.datetime.fromisoformat(
context.resolve_value(raw_timestamp, as_type=str),
)
class MyClass(dg.Resolvable, dg.Model):
event: str
# the yaml field will be a string, which is then parsed into a datetime object
timestamp: Annotated[
datetime.datetime,
dg.Resolver(resolve_timestamp, model_field_type=str),
]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Model'>`class` dagster.Model <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/resolved/model.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Model" class="hash-link"></a></Link></dt>
<dd>
pydantic BaseModel configured with recommended default settings for use with the Resolved framework.
Extra fields are disallowed when instantiating this model to help catch errors earlier.
Example:
```python
import dagster as dg
class MyModel(dg.Resolvable, dg.Model):
name: str
age: int
# raises exception
MyModel(name="John", age=30, other="field")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.template_var'>@dagster.template_var <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/component/template_vars.py#L62' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.template_var" class="hash-link"></a></Link></dt>
<dd>
Decorator that marks a function as a template variable for use in component YAML definitions.
Template variables provide dynamic values and functions that can be injected into component
YAML definitions using Jinja2 templating syntax (\{\{ variable_name }}). They are evaluated
at component load time and can optionally receive a ComponentLoadContext parameter for
context-aware behavior.
These values can be any python object and are passed directly to the component as Python object.
They can be injected at any level of the defs file.
There are two main usage patterns:
1. <strong>Module-level template variables</strong>: Functions defined in a separate module and referenced via the `template_vars_module` field in component YAML
2. <strong>Component class static methods</strong>: Template variables defined as `@staticmethod` on a Component class, automatically available to instances of that component
Template vars can themselves be functions, in which case they are user-defined functions, invoked
with function syntax within the defs file.
Parameters: <strong>fn</strong> – The function to decorate as a template variable. If None, returns a decorator.Returns: The decorated function with template variable metadata, or a decorator function.
Note: Template variables are evaluated at component load time, not at runtime. They provide
configuration values and functions for YAML templating, not runtime component logic.
Function Signatures:
Template variable functions can have one of two valid signatures:
<strong>Zero parameters (static values)</strong>:
```python
@dg.template_var
def static_value() -> Any:
# Returns a static value computed at load time
return "computed_value"
```
<strong>Single ComponentLoadContext parameter (context-aware)</strong>:
```python
@dg.template_var
def context_value(context: dg.ComponentLoadContext) -> Any:
# Returns a value based on the component's loading context
return f"value_{context.path.name}"
```
<strong>Return Types:</strong>
Template variables can return any type, including:
- <strong>Primitive values</strong>: `str`, `int`, `bool`, `float`
- <strong>Collections</strong>: `list`, `dict`, `set`, `tuple`
- <strong>Complex objects</strong>: `PartitionsDefinition`, custom classes, etc.
- <strong>Functions</strong>: `Callable` objects for use as UDFs in Jinja2 templates
<strong>Invalid Signatures:</strong>
```python
# ❌ Multiple parameters not allowed
@dg.template_var
def invalid_multiple_params(context: ComponentLoadContext, other_param: str):
pass
# ❌ Wrong context type
@dg.template_var
def invalid_context_type(context: ComponentDeclLoadContext):
pass
# ❌ Static methods with parameters other than context
class MyComponent(dg.Component):
@staticmethod
@dg.template_var
def invalid_static(param: str): # Only 0 or 1 (context) params allowed
pass
```
Examples:
<strong>Basic template variable (no context needed)</strong>:
```python
import dagster as dg
import os
@dg.template_var
def database_url() -> str:
if os.getenv("ENVIRONMENT") == "prod":
return "postgresql://prod-server:5432/db"
else:
return "postgresql://localhost:5432/dev_db"
```
<strong>Context-aware template variable</strong>:
```python
@dg.template_var
def component_specific_table(context: dg.ComponentLoadContext) -> str:
return f"table_{context.path.name}"
```
<strong>Template variable returning a function</strong>:
This is colloquially called a “udf” (user-defined function).
```python
@dg.template_var
def table_name_generator() -> Callable[[str], str]:
return lambda prefix: f"{prefix}_processed_data"
```
<strong>Using template variables in YAML</strong>:
```yaml
# defs.yaml
type: my_project.components.DataProcessor
template_vars_module: .template_vars
attributes:
database_url: "{{ database_url }}"
table_name: "{{ component_specific_table }}"
processed_table: "{{ table_name_generator('sales') }}"
```
<strong>Component class static methods</strong>:
```python
class MyComponent(dg.Component):
@staticmethod
@dg.template_var
def default_config() -> dict:
return {"timeout": 30, "retries": 3}
@staticmethod
@dg.template_var
def context_aware_value(context: dg.ComponentLoadContext) -> str:
return f"value_for_{context.path.name}"
```
<strong>Using in YAML (component static methods)</strong>:
```yaml
type: my_project.components.MyComponent
attributes:
config: "{{ default_config }}"
name: "{{ context_aware_value }}"
```
See also: - [`dagster.ComponentLoadContext`](#dagster.ComponentLoadContext): Context object available to template variables
</dd>
</dl>
<div class="section" id="core-models">
### Core Models
These Annotated TypeAliases can be used when defining custom Components for
common Dagster types.
<dl>
<dt><Link class="anchor" id='dagster.ResolvedAssetKey'>dagster.ResolvedAssetKey`:` Annotated`[`[AssetKey](assets.mdx#dagster.AssetKey)`,` `...``]` <a href='https://github.com/dagster-io/dagster/blob/master/uv/python/versions/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/typing.py#L1285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResolvedAssetKey" class="hash-link"></a></Link></dt>
<dd>
Allows resolving to an AssetKey via a YAML-friendly schema.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResolvedAssetSpec'>dagster.ResolvedAssetSpec`:` Annotated`[`[AssetSpec](assets.mdx#dagster.AssetSpec)`,` `...``]` <a href='https://github.com/dagster-io/dagster/blob/master/uv/python/versions/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/typing.py#L1285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResolvedAssetSpec" class="hash-link"></a></Link></dt>
<dd>
Allows resolving to an AssetSpec via a YAML-friendly schema.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetAttributesModel'>dagster.AssetAttributesModel<a href="#dagster.AssetAttributesModel" class="hash-link"></a></Link></dt>
<dd>
A pydantic modeling of all the attributes of an AssetSpec that can be set before the definition is created.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResolvedAssetCheckSpec'>dagster.ResolvedAssetCheckSpec`:` Annotated`[`[AssetCheckSpec](asset-checks.mdx#dagster.AssetCheckSpec)`,` `...``]` <a href='https://github.com/dagster-io/dagster/blob/master/uv/python/versions/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/typing.py#L1285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResolvedAssetCheckSpec" class="hash-link"></a></Link></dt>
<dd>
Allows resolving to an AssetCheckSpec via a YAML-friendly schema.
</dd>
</dl>
<div class="section" id="built-in-components">
#### Built-in Components
<dl>
<dt><Link class="anchor" id='dagster.DefsFolderComponent'>`class` dagster.DefsFolderComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/core/defs_module.py#L305' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DefsFolderComponent" class="hash-link"></a></Link></dt>
<dd>
A component that represents a directory containing multiple Dagster definition modules.
DefsFolderComponent serves as a container for organizing and managing multiple subcomponents
within a folder structure. It automatically discovers and loads components from subdirectories
and files, enabling hierarchical organization of Dagster definitions. This component also
supports post-processing capabilities to modify metadata and properties of definitions
created by its child components.
Key Features:
- <strong>Post-Processing</strong>: Allows modification of child component definitions via configuration
- <strong>Automatic Discovery</strong>: Recursively finds and loads components from subdirectories
- <strong>Hierarchical Organization</strong>: Enables nested folder structures for complex projects
The component automatically scans its directory for:
- YAML component definitions (`defs.yaml` files)
- Python modules containing Dagster definitions
- Nested subdirectories containing more components
Here is how a DefsFolderComponent is used in a project by the framework, along
with other framework-defined classes.
```text
my_project/
└── defs/
├── analytics/ # DefsFolderComponent
│ ├── defs.yaml # Post-processing configuration
│ ├── user_metrics/ # User-defined component
│ │ └── defs.yaml
│ └── sales_reports/ # User-defined component
│ └── defs.yaml
└── data_ingestion/ # DefsFolderComponent
├── api_sources/ # DefsFolderComponent
│ └── some_defs.py # PythonFileComponent
└── file_sources/ # DefsFolderComponent
└── files.py # PythonFileComponent
```
Parameters:
- <strong>path</strong> – The filesystem path to the directory containing child components.
- <strong>children</strong> – A mapping of child paths to their corresponding Component instances. This is typically populated automatically during component discovery.
DefsFolderComponent supports post-processing through its `defs.yaml` configuration,
allowing you to modify definitions created by child components using target selectors
Examples:
Using post-processing in a folder’s `defs.yaml`:
```yaml
# analytics/defs.yaml
type: dagster.DefsFolderComponent
post_processing:
assets:
- target: "*" # add a top level tag to all assets in the folder
attributes:
tags:
top_level_tag: "true"
- target: "tag:defs_tag=true" # add a tag to all assets in the folder with the tag "defs_tag"
attributes:
tags:
new_tag: "true"
```
Please see documentation on post processing and the selection syntax for more examples.
Component Discovery:
The component automatically discovers children using these patterns:
1. <strong>YAML Components</strong>: Subdirectories with `defs.yaml` files
2. <strong>Python Modules</strong>: Any `.py` files containing Dagster definitions
3. <strong>Nested Folders</strong>: Subdirectories that contain any of the above
Files and directories matching these patterns are ignored:
- `__pycache__` directories
- Hidden directories (starting with `.`)
Note: DefsFolderComponent instances are typically created automatically by Dagster’s
component loading system. Manual instantiation is rarely needed unless building
custom loading logic or testing scenarios.
When used with post-processing, the folder’s `defs.yaml` should only contain
post-processing configuration, not component type definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DefinitionsComponent'>`class` dagster.DefinitionsComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/definitions_component/__init__.py#L16' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DefinitionsComponent" class="hash-link"></a></Link></dt>
<dd>
An arbitrary set of Dagster definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.UvRunComponent'>`class` dagster.UvRunComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/executable_component/uv_run_component.py#L25' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.UvRunComponent" class="hash-link"></a></Link></dt>
<dd>
Represents a Python script, alongside the set of assets or asset checks that it is responsible for executing.
Accepts a path to a Python script which will be executed in a dagster-pipes subprocess using the <cite>uv run</cite> command.
Example:
```yaml
type: dagster.UvRunComponent
attributes:
execution:
path: update_table.py
assets:
- key: my_table
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PythonScriptComponent'>`class` dagster.PythonScriptComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/executable_component/python_script_component.py#L25' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PythonScriptComponent" class="hash-link"></a></Link></dt>
<dd>
Represents a Python script, alongside the set of assets and asset checks that it is responsible for executing.
Accepts a path to a Python script which will be executed in a dagster-pipes subprocess using your installed <cite>python</cite> executable.
Examples:
```yaml
type: dagster.PythonScriptComponent
attributes:
execution:
path: update_table.py
assets:
- key: my_table
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FunctionComponent'>`class` dagster.FunctionComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/executable_component/function_component.py#L95' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FunctionComponent" class="hash-link"></a></Link></dt>
<dd>
Represents a Python function, alongside the set of assets or asset checks that it is responsible for executing.
The provided function should return either a <cite>MaterializeResult</cite> or an <cite>AssetCheckResult</cite>.
Examples:
```yaml
type: dagster.FunctionComponent
attributes:
execution:
fn: .my_module.update_table
assets:
- key: my_table
```
```python
from dagster import MaterializeResult
def update_table(context: AssetExecutionContext) -> MaterializeResult:
# ...
return MaterializeResult(metadata={"rows_updated": 100})
@component
def my_component():
return FunctionComponent(
execution=update_table,
assets=[AssetSpec(key="my_table")],
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SqlComponent'>`class` dagster.SqlComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/sql_component/sql_component.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SqlComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Base component which executes templated SQL. Subclasses
implement instructions on where to load the SQL content from.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TemplatedSqlComponent'>`class` dagster.TemplatedSqlComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/lib/sql_component/sql_component.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TemplatedSqlComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component which executes templated SQL from a string or file.
</dd>
</dl>
</div>
<div class="section" id="testing-components">
#### Testing Components
<dl>
<dt><Link class="anchor" id='dagster.components.testing.create_defs_folder_sandbox'>dagster.components.testing.create_defs_folder_sandbox <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/testing/utils.py#L291' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.components.testing.create_defs_folder_sandbox" class="hash-link"></a></Link></dt>
<dd>
Create a lightweight sandbox to scaffold and instantiate components. Useful
for those authoring custom components.
This function creates a temporary project that mimics the `defs` folder portion
of a real Dagster project. It then yields a [`DefsFolderSandbox`](#dagster.components.testing.DefsFolderSandbox) object which can be used to
scaffold and load components.
[`DefsFolderSandbox`](#dagster.components.testing.DefsFolderSandbox) has a few properties useful for different types of tests:
- `defs_folder_path`: The absolute path to the `defs` folder. The user can inspect and load files from scaffolded components, e.g. `(defs_folder_path / "my_component" / "defs.yaml").exists()`
- `project_name`: If not provided, a random name is generated.
Once the sandbox is created, you can load all definitions using the `load` method on [`DefsFolderSandbox`](#dagster.components.testing.DefsFolderSandbox), or with the `load_component` method.
This sandbox does not provide complete environmental isolation, but does provide some isolation guarantees
to do its best to isolate the test from and restore the environment after the test.
- A file structure like this is created: `\<\<temp folder>> / src / \<\<project_name>> / defs`
- `\<\<temp folder>> / src` is placed in `sys.path` during the loading process
- Any modules loaded during the process that descend from defs module are evicted from `sys.modules` on cleanup.
Parameters: <strong>project_name</strong> – Optional name for the project (default: random name).Returns: A context manager that yields a DefsFolderSandboxReturn type: Iterator[[DefsFolderSandbox](#dagster.components.testing.DefsFolderSandbox)]
Example:
```python
with create_defs_folder_sandbox() as sandbox:
defs_path = sandbox.scaffold_component(component_cls=MyComponent)
assert (defs_path / "defs.yaml").exists()
assert (defs_path / "my_component_config_file.yaml").exists() # produced by MyComponentScaffolder
with create_defs_folder_sandbox() as sandbox:
defs_path = sandbox.scaffold_component(
component_cls=MyComponent,
defs_yaml_contents={"type": "MyComponent", "attributes": {"asset_key": "my_asset"}},
)
with sandbox.load_component_and_build_defs(defs_path=defs_path) as (component, defs):
assert isinstance(component, MyComponent)
assert defs.get_asset_def("my_asset").key == AssetKey("my_asset")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.components.testing.DefsFolderSandbox'>`class` dagster.components.testing.DefsFolderSandbox <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/testing/utils.py#L124' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.components.testing.DefsFolderSandbox" class="hash-link"></a></Link></dt>
<dd>
A sandbox for testing components.
This sandbox provides a number of utilities for scaffolding, modifying, and loading components
from a temporary defs folder. This makes it easy to test components in isolation.
<dl>
<dt><Link class="anchor" id='dagster.components.testing.DefsFolderSandbox.load_component_and_build_defs'>load_component_and_build_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/testing/utils.py#L166' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.components.testing.DefsFolderSandbox.load_component_and_build_defs" class="hash-link"></a></Link></dt>
<dd>
Loads a Component object at the given path and builds the corresponding Definitions.
Parameters: <strong>defs_path</strong> – The path to the component to load.Returns: A tuple of the Component and Definitions objects.
Example:
```python
with scaffold_defs_sandbox() as sandbox:
defs_path = sandbox.scaffold_component(component_cls=MyComponent)
with sandbox.load_component_and_build_defs(defs_path=defs_path) as (
component,
defs,
):
assert isinstance(component, MyComponent)
assert defs.get_asset_def("my_asset").key == AssetKey("my_asset")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.components.testing.DefsFolderSandbox.scaffold_component'>scaffold_component <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/testing/utils.py#L197' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.components.testing.DefsFolderSandbox.scaffold_component" class="hash-link"></a></Link></dt>
<dd>
Scaffolds a component into the defs folder.
Parameters:
- <strong>component_cls</strong> – The component class to scaffold.
- <strong>defs_path</strong> – The path to the component. (defaults to a random name)
- <strong>scaffold_params</strong> – The parameters to pass to the scaffolder.
- <strong>scaffold_format</strong> – The format to use for scaffolding.
- <strong>defs_yaml_contents</strong> – The body of the component to update the defs.yaml file with.
Returns: The path to the component.
Example:
```python
with scaffold_defs_sandbox() as sandbox:
defs_path = sandbox.scaffold_component(component_cls=MyComponent)
assert (defs_path / "defs.yaml").exists()
```
</dd>
</dl>
</dd>
</dl>
</div></div></div>
<div class="section" id="using-components">
## Using Components
<dl>
<dt><Link class="anchor" id='dagster.component_instance'>@dagster.component_instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/component/component_loader.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.component_instance" class="hash-link"></a></Link></dt>
<dd>
Decorator for a function to be used to load an instance of a Component.
This is used when instantiating components in python instead of via yaml.
Example:
```python
import dagster as dg
class MyComponent(dg.Component):
...
@dg.component_instance
def load(context: dg.ComponentLoadContext) -> MyComponent:
return MyComponent(...)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ComponentLoadContext'>`class` dagster.ComponentLoadContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/core/context.py#L238' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ComponentLoadContext" class="hash-link"></a></Link></dt>
<dd>
Context object that provides environment and path information during component loading.
This context is automatically created and passed to component definitions when loading
a project’s defs folder. Each Python module or folder in the defs directory receives
a unique context instance that provides access to the underlying ComponentDecl,
project structure, paths, and utilities for dynamic component instantiation.
The context enables components to:
- Access project and module path information
- Load other modules and definitions within the project
- Resolve relative imports and module names
- Access templating and resolution capabilities
Parameters:
- <strong>path</strong> – The filesystem path of the component currently being loaded. For a file: `/path/to/project/src/project/defs/my_component.py` For a directory: `/path/to/project/src/project/defs/my_component/`
- <strong>project_root</strong> – The root directory of the Dagster project, typically containing `pyproject.toml` or `setup.py`. Example: `/path/to/project`
- <strong>defs_module_path</strong> – The filesystem path to the root defs folder. Example: `/path/to/project/src/project/defs`
- <strong>defs_module_name</strong> – The Python module name for the root defs folder, used for import resolution. Typically follows the pattern `"project_name.defs"`. Example: `"my_project.defs"`
- <strong>resolution_context</strong> – The resolution context used by the component templating system for parameter resolution and variable substitution.
- <strong>component_tree</strong> – The component tree that contains the component currently being loaded.
- <strong>terminate_autoloading_on_keyword_files</strong> – Controls whether autoloading stops when encountering `definitions.py` or `component.py` files. <strong>Deprecated</strong>: This parameter will be removed after version 1.11.
- <strong>component_decl</strong> – The associated ComponentDecl to the component being loaded.
Note: This context is automatically provided by Dagster’s autoloading system and
should not be instantiated manually in most cases. For testing purposes,
use `ComponentTree.for_test().load_context` to create a test instance.
See also: - [`dagster.definitions()`](definitions.mdx#dagster.definitions): Decorator that receives this context
- [`dagster.Definitions`](definitions.mdx#dagster.Definitions): The object typically returned by context-using functions
- [`dagster.components.resolved.context.ResolutionContext`](#dagster.ResolutionContext): Underlying resolution context
- `dagster.ComponentDeclLoadContext`: Context available when loading ComponentDecls
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ComponentTree'>`class` dagster.ComponentTree <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/core/component_tree.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ComponentTree" class="hash-link"></a></Link></dt>
<dd>
The hierarchy of Component instances defined in the project.
Manages and caches the component loading process, including finding component declarations
to build the initial declaration tree, loading these Components, and eventually building the
Definitions.
</dd>
</dl>
</div>
<div class="section" id="loading-component-definitions">
## Loading Component definitions
<dl>
<dt><Link class="anchor" id='dagster.load_from_defs_folder'>dagster.load_from_defs_folder <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/core/load_defs.py#L128' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_from_defs_folder" class="hash-link"></a></Link></dt>
<dd>
Constructs a Definitions object by automatically discovering and loading all Dagster
definitions from a project’s defs folder structure.
This function serves as the primary entry point for loading definitions in dg-managed
projects. It reads the project configuration (dg.toml or pyproject.toml), identifies
the defs module, and recursively loads all components, assets, jobs, and other Dagster
definitions from the project structure.
The function automatically handles:
- Reading project configuration to determine the defs module location
- Importing and traversing the defs module hierarchy
- Loading component definitions and merging them into a unified Definitions object
- Enriching definitions with plugin component metadata from entry points
Parameters: <strong>path_within_project</strong> (<em>Path</em>) – A path within the dg project directory.
This directory or a parent of should contain the project’s configuration file
(dg.toml or pyproject.toml with [tool.dg] section).Returns:
A merged Definitions object containing all discovered definitions
from the project’s defs folder, enriched with component metadata.
Return type: [Definitions](definitions.mdx#dagster.Definitions)
Example:
```python
from pathlib import Path
import dagster as dg
@dg.definitions
def defs():
project_path = Path("/path/to/my/dg/project")
return dg.load_from_defs_folder(project_root=project_path)
```
</dd>
</dl>
</div></div>
---
---
title: 'config'
sidebar_position: 1000
title_meta: 'config API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'config Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="config">
# Config
<div class="section" id="pythonic-config-system">
## Pythonic config system
The following classes are used as part of the new [Pythonic config system](https://docs.dagster.io/guides/operate/configuration/advanced-config-types). They are used in conjunction with builtin types.
<dl>
<dt><Link class="anchor" id='dagster.Config'>`class` dagster.Config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/pythonic_config/config.py#L155' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Config" class="hash-link"></a></Link></dt>
<dd>
Base class for Dagster configuration models, used to specify config schema for
ops and assets. Subclasses `pydantic.BaseModel`.
Example definition:
```python
from pydantic import Field
class MyAssetConfig(Config):
my_str: str = "my_default_string"
my_int_list: List[int]
my_bool_with_metadata: bool = Field(default=False, description="A bool field")
```
Example usage:
```python
@asset
def asset_with_config(config: MyAssetConfig):
assert config.my_str == "my_default_string"
assert config.my_int_list == [1, 2, 3]
assert config.my_bool_with_metadata == False
asset_with_config(MyAssetConfig(my_int_list=[1, 2, 3], my_bool_with_metadata=True))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PermissiveConfig'>`class` dagster.PermissiveConfig <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/pythonic_config/config.py#L379' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PermissiveConfig" class="hash-link"></a></Link></dt>
<dd>
Subclass of [`Config`](#dagster.Config) that allows arbitrary extra fields. This is useful for
config classes which may have open-ended inputs.
Example definition:
```python
class MyPermissiveOpConfig(PermissiveConfig):
my_explicit_parameter: bool
my_other_explicit_parameter: str
```
Example usage:
```python
@op
def op_with_config(config: MyPermissiveOpConfig):
assert config.my_explicit_parameter == True
assert config.my_other_explicit_parameter == "foo"
assert config.dict().get("my_implicit_parameter") == "bar"
op_with_config(
MyPermissiveOpConfig(
my_explicit_parameter=True,
my_other_explicit_parameter="foo",
my_implicit_parameter="bar"
)
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunConfig'>`class` dagster.RunConfig <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_config.py#L629' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunConfig" class="hash-link"></a></Link></dt>
<dd>
Container for all the configuration that can be passed to a run. Accepts Pythonic definitions
for op and asset config and resources and converts them under the hood to the appropriate config dictionaries.
Example usage:
```python
class MyAssetConfig(Config):
a_str: str
@asset
def my_asset(config: MyAssetConfig):
assert config.a_str == "foo"
materialize(
[my_asset],
run_config=RunConfig(
ops={"my_asset": MyAssetConfig(a_str="foo")}
)
)
```
<dl>
<dt><Link class="anchor" id='dagster.RunConfig.to_config_dict'>to_config_dict <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_config.py#L666' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunConfig.to_config_dict" class="hash-link"></a></Link></dt>
<dd>
Converts the RunConfig to a dictionary representation.
Returns: The dictionary representation of the RunConfig.Return type: Dict[str, Any]
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="legacy-dagster-config-types">
## Legacy Dagster config types
The following types are used as part of the legacy [Dagster config system](https://docs.dagster.io/guides/operate/configuration). They are used in conjunction with builtin types.
<dl>
<dt><Link class="anchor" id='dagster.ConfigSchema'>`class` dagster.ConfigSchema <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_schema.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ConfigSchema" class="hash-link"></a></Link></dt>
<dd>
Placeholder type for config schemas.
Any time that it appears in documentation, it means that any of the following types are
acceptable:
1. A Python scalar type that resolves to a Dagster config type (`python:int`, `python:float`, `python:bool`, or `python:str`). For example:
- `@op(config_schema=int)`
- `@op(config_schema=str)`
2. A built-in python collection (`python:list`, or `python:dict`). `python:list` is exactly equivalent to [`Array`](#dagster.Array) [ `Any` ] and `python:dict` is equivalent to [`Permissive`](#dagster.Permissive). For example:
- `@op(config_schema=list)`
- `@op(config_schema=dict)`
3. A Dagster config type:
- `Any`
- [`Array`](#dagster.Array)
- `Bool`
- [`Enum`](#dagster.Enum)
- `Float`
- `Int`
- [`IntSource`](#dagster.IntSource)
- [`Noneable`](#dagster.Noneable)
- [`Permissive`](#dagster.Permissive)
- [`Map`](#dagster.Map)
- [`ScalarUnion`](#dagster.ScalarUnion)
- [`Selector`](#dagster.Selector)
- [`Shape`](#dagster.Shape)
- `String`
- [`StringSource`](#dagster.StringSource)
4. A bare python dictionary, which will be automatically wrapped in [`Shape`](#dagster.Shape). Values of the dictionary are resolved recursively according to the same rules. For example:
- `\{'some_config': str}` is equivalent to `Shape(\{'some_config: str})`.
- `\{'some_config1': \{'some_config2': str}}` is equivalent to
5. A bare python list of length one, whose single element will be wrapped in a [`Array`](#dagster.Array) is resolved recursively according to the same rules. For example:
- `[str]` is equivalent to `Array[str]`.
- `[[str]]` is equivalent to `Array[Array[str]]`.
- `[\{'some_config': str}]` is equivalent to `Array(Shape(\{'some_config: str}))`.
6. An instance of [`Field`](#dagster.Field).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Field'>`class` dagster.Field <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L181' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field" class="hash-link"></a></Link></dt>
<dd>
Defines the schema for a configuration field.
Fields are used in config schema instead of bare types when one wants to add a description,
a default value, or to mark it as not required.
Config fields are parsed according to their schemas in order to yield values available at
job execution time through the config system. Config fields can be set on ops, on
loaders for custom, and on other pluggable components of the system, such as resources, loggers,
and executors.
Parameters:
- <strong>config</strong> (<em>Any</em>) –
The schema for the config. This value can be any of:
1. A Python primitive type that resolves to a Dagster config type (`python:int`, `python:float`, `python:bool`, `python:str`, or `python:list`).
2. A Dagster config type:
- `Any`
- [`Array`](#dagster.Array)
- `Bool`
- [`Enum`](#dagster.Enum)
- `Float`
- `Int`
- [`IntSource`](#dagster.IntSource)
- [`Noneable`](#dagster.Noneable)
- [`Permissive`](#dagster.Permissive)
- [`ScalarUnion`](#dagster.ScalarUnion)
- [`Selector`](#dagster.Selector)
- [`Shape`](#dagster.Shape)
- `String`
- [`StringSource`](#dagster.StringSource)
3. A bare python dictionary, which will be automatically wrapped in [`Shape`](#dagster.Shape). Values of the dictionary are resolved recursively according to the same rules.
4. A bare python list of length one which itself is config type. Becomes [`Array`](#dagster.Array) with list element as an argument.
- <strong>default_value</strong> (<em>Any</em>) –
A default value for this field, conformant to the schema set by the `dagster_type` argument. If a default value is provided, `is_required` should be `False`.
- <strong>is_required</strong> (<em>bool</em>) – Whether the presence of this field is required. Defaults to true. If `is_required` is `True`, no default value should be provided.
- <strong>description</strong> (<em>str</em>) – A human-readable description of this config field.
- <strong>is_secret</strong> (<em>bool</em>) – Whether this field contains sensitive data that should be masked in UIs. Defaults to False.
Examples:
```python
@op(
config_schema={
'word': Field(str, description='I am a word.'),
'repeats': Field(Int, default_value=1, is_required=False),
}
)
def repeat_word(context):
return context.op_config['word'] * context.op_config['repeats']
```
<dl>
<dt><Link class="anchor" id='dagster.Field.default_provided'>`property` default_provided <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L349' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field.default_provided" class="hash-link"></a></Link></dt>
<dd>
Was a default value provided.
Returns: Yes or noReturn type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Field.default_value'>`property` default_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L359' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field.default_value" class="hash-link"></a></Link></dt>
<dd>
The default value for the field.
Raises an exception if no default value was provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Field.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L369' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field.description" class="hash-link"></a></Link></dt>
<dd>
A human-readable description of this config field, if provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Field.is_required'>`property` is_required <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L340' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field.is_required" class="hash-link"></a></Link></dt>
<dd>
Whether a value for this field must be provided at runtime.
Cannot be True if a default value is provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Field.is_secret'>`property` is_secret <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field.py#L375' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Field.is_secret" class="hash-link"></a></Link></dt>
<dd>
Whether this field contains sensitive data that should be masked in UIs.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Selector'>`class` dagster.Selector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L277' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Selector" class="hash-link"></a></Link></dt>
<dd>
Define a config field requiring the user to select one option.
Selectors are used when you want to be able to present several different options in config but
allow only one to be selected. For example, a single input might be read in from either a csv
file or a parquet file, but not both at once.
Note that in some other type systems this might be called an ‘input union’.
Functionally, a selector is like a `Dict`, except that only one key from the dict can
be specified in valid config.
Parameters: <strong>fields</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em>[*Field*](#dagster.Field)<em>]</em>) – The fields from which the user must select.
<strong>Examples:</strong>
```python
@op(
config_schema=Field(
Selector(
{
'haw': {'whom': Field(String, default_value='honua', is_required=False)},
'cn': {'whom': Field(String, default_value='世界', is_required=False)},
'en': {'whom': Field(String, default_value='world', is_required=False)},
}
),
is_required=False,
default_value={'en': {'whom': 'world'}},
)
)
def hello_world_with_default(context):
if 'haw' in context.op_config:
return 'Aloha {whom}!'.format(whom=context.op_config['haw']['whom'])
if 'cn' in context.op_config:
return '你好, {whom}!'.format(whom=context.op_config['cn']['whom'])
if 'en' in context.op_config:
return 'Hello, {whom}!'.format(whom=context.op_config['en']['whom'])
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Permissive'>`class` dagster.Permissive <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L229' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Permissive" class="hash-link"></a></Link></dt>
<dd>
Defines a config dict with a partially specified schema.
A permissive dict allows partial specification of the config schema. Any fields with a
specified schema will be type checked. Other fields will be allowed, but will be ignored by
the type checker.
Parameters: <strong>fields</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em>[*Field*](#dagster.Field)<em>]</em>) – The partial specification of the config dict.
<strong>Examples:</strong>
```python
@op(config_schema=Field(Permissive({'required': Field(String)})))
def map_config_op(context) -> List:
return sorted(list(context.op_config.items()))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Shape'>`class` dagster.Shape <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L105' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Shape" class="hash-link"></a></Link></dt>
<dd>
Schema for configuration data with string keys and typed values via [`Field`](#dagster.Field).
Unlike [`Permissive`](#dagster.Permissive), unspecified fields are not allowed and will throw a
[`DagsterInvalidConfigError`](errors.mdx#dagster.DagsterInvalidConfigError).
Parameters:
- <strong>fields</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em>[*Field*](#dagster.Field)<em>]</em>) – The specification of the config dict.
- <strong>field_aliases</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em>) – Maps a string key to an alias that can be used instead of the original key. For example, an entry \{“foo”: “bar”} means that someone could use “bar” instead of “foo” as a top level string key.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Map'>`class` dagster.Map <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L156' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Map" class="hash-link"></a></Link></dt>
<dd>
Defines a config dict with arbitrary scalar keys and typed values.
A map can contrain arbitrary keys of the specified scalar type, each of which has
type checked values. Unlike [`Shape`](#dagster.Shape) and [`Permissive`](#dagster.Permissive), scalar
keys other than strings can be used, and unlike [`Permissive`](#dagster.Permissive), all
values are type checked.
Parameters:
- <strong>key_type</strong> (<em>type</em>) – The type of keys this map can contain. Must be a scalar type.
- <strong>inner_type</strong> (<em>type</em>) – The type of the values that this map type can contain.
- <strong>key_label_name</strong> (<em>string</em>) – Optional name which describes the role of keys in the map.
<strong>Examples:</strong>
```python
@op(config_schema=Field(Map({str: int})))
def partially_specified_config(context) -> List:
return sorted(list(context.op_config.items()))
```
<dl>
<dt><Link class="anchor" id='dagster.Map.key_label_name'>`property` key_label_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L209' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Map.key_label_name" class="hash-link"></a></Link></dt>
<dd>
Name which describes the role of keys in the map, if provided.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Array'>`class` dagster.Array <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L216' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Array" class="hash-link"></a></Link></dt>
<dd>
Defines an array (list) configuration type that contains values of type `inner_type`.
Parameters: <strong>inner_type</strong> (<em>type</em>) – The type of the values that this configuration type can contain.
<dl>
<dt><Link class="anchor" id='dagster.Array.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L235' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Array.description" class="hash-link"></a></Link></dt>
<dd>
A human-readable description of this Array type.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Noneable'>`class` dagster.Noneable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L182' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Noneable" class="hash-link"></a></Link></dt>
<dd>
Defines a configuration type that is the union of `NoneType` and the type `inner_type`.
Parameters: <strong>inner_type</strong> (<em>type</em>) – The type of the values that this configuration type can contain.
<strong>Examples:</strong>
```python
config_schema={"name": Noneable(str)}
config={"name": "Hello"} # Ok
config={"name": None} # Ok
config={} # Error
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Enum'>`class` dagster.Enum <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L271' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Enum" class="hash-link"></a></Link></dt>
<dd>
Defines a enum configuration type that allows one of a defined set of possible values.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the enum configuration type.
- <strong>enum_values</strong> (<em>List</em><em>[</em>[*EnumValue*](#dagster.EnumValue)<em>]</em>) – The set of possible values for the enum configuration type.
<strong>Examples:</strong>
```python
from dagster import Field, op
from dagster._config.config_type import Enum, EnumValue
@op(
config_schema=Field(
Enum(
'CowboyType',
[
EnumValue('good'),
EnumValue('bad'),
EnumValue('ugly'),
]
)
)
)
def resolve_standoff(context):
# Implementation here
pass
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EnumValue'>`class` dagster.EnumValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L246' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EnumValue" class="hash-link"></a></Link></dt>
<dd>
Define an entry in a [`Enum`](#dagster.Enum).
Parameters:
- <strong>config_value</strong> (<em>str</em>) – The string representation of the config to accept when passed.
- <strong>python_value</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The python value to convert the enum entry in to. Defaults to the `config_value`.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the enum entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScalarUnion'>`class` dagster.ScalarUnion <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/config_type.py#L390' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScalarUnion" class="hash-link"></a></Link></dt>
<dd>
Defines a configuration type that accepts a scalar value OR a non-scalar value like a
`List`, `Dict`, or [`Selector`](#dagster.Selector).
This allows runtime scalars to be configured without a dictionary with the key `value` and
instead just use the scalar value directly. However this still leaves the option to
load scalars from a json or pickle file.
Parameters:
- <strong>scalar_type</strong> (<em>type</em>) – The scalar type of values that this configuration type can hold. For example, `python:int`, `python:float`, `python:bool`, or `python:str`.
- <strong>non_scalar_schema</strong> ([*ConfigSchema*](#dagster.ConfigSchema)) – The schema of a non-scalar Dagster configuration type. For example, `List`, `Dict`, or [`Selector`](#dagster.Selector).
- <strong>key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The configuation type’s unique key. If not set, then the key will be set to `ScalarUnion.\{scalar_type}-\{non_scalar_schema}`.
<strong>Examples:</strong>
```yaml
graph:
transform_word:
inputs:
word:
value: foobar
```
becomes, optionally,
```yaml
graph:
transform_word:
inputs:
word: foobar
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StringSource'>dagster.StringSource<a href="#dagster.StringSource" class="hash-link"></a></Link></dt>
<dd>
Use this type when you want to read a string config value from an environment variable. The value
passed to a config field of this type may either be a string literal, or a selector describing
how to look up the value from the executing process’s environment variables.
<strong>Examples:</strong>
```python
from dagster import job, op, StringSource
@op(config_schema=StringSource)
def secret_op(context) -> str:
return context.op_config
@job
def secret_job():
secret_op()
secret_job.execute_in_process(
run_config={
'ops': {'secret_op': {'config': 'test_value'}}
}
)
secret_job.execute_in_process(
run_config={
'ops': {'secret_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE'}}}
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IntSource'>dagster.IntSource<a href="#dagster.IntSource" class="hash-link"></a></Link></dt>
<dd>
Use this type when you want to read an integer config value from an environment variable. The
value passed to a config field of this type may either be a integer literal, or a selector
describing how to look up the value from the executing process’s environment variables.
<strong>Examples:</strong>
```python
from dagster import job, op, IntSource
@op(config_schema=IntSource)
def secret_int_op(context) -> int:
return context.op_config
@job
def secret_job():
secret_int_op()
secret_job.execute_in_process(
run_config={
'ops': {'secret_int_op': {'config': 1234}}
}
)
secret_job.execute_in_process(
run_config={
'ops': {'secret_int_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_INT'}}}
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.BoolSource'>dagster.BoolSource<a href="#dagster.BoolSource" class="hash-link"></a></Link></dt>
<dd>
Use this type when you want to read an boolean config value from an environment variable. The
value passed to a config field of this type may either be a boolean literal, or a selector
describing how to look up the value from the executing process’s environment variables. Set the
value of the corresponding environment variable to `""` to indicate `False`.
<strong>Examples:</strong>
```python
from dagster import job, op, BoolSource
@op(config_schema=BoolSource)
def secret_bool_op(context) -> bool:
return context.op_config
@job
def secret_job():
secret_bool_op()
secret_job.execute_in_process(
run_config={
'ops': {'secret_bool_op': {'config': False}}
}
)
secret_job.execute_in_process(
run_config={
'ops': {'secret_bool_op': {'config': {'env': 'VERY_SECRET_ENV_VARIABLE_BOOL'}}}
}
)
```
</dd>
</dl>
</div>
<div class="section" id="config-utilities">
## Config Utilities
<dl>
<dt><Link class="anchor" id='dagster.ConfigMapping'>`class` dagster.ConfigMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/config.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ConfigMapping" class="hash-link"></a></Link></dt>
<dd>
Defines a config mapping for a graph (or job).
By specifying a config mapping function, you can override the configuration for the child
ops and graphs contained within a graph.
Config mappings require the configuration schema to be specified as `config_schema`, which will
be exposed as the configuration schema for the graph, as well as a configuration mapping
function, `config_fn`, which maps the config provided to the graph to the config
that will be provided to the child nodes.
Parameters:
- <strong>config_fn</strong> (<em>Callable</em><em>[</em><em>[</em><em>dict</em><em>]</em><em>, </em><em>dict</em><em>]</em>) – The function that will be called to map the graph config to a config appropriate for the child nodes.
- <strong>config_schema</strong> ([*ConfigSchema*](#dagster.ConfigSchema)) – The schema of the graph config.
- <strong>receive_processed_config_values</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If true, config values provided to the config_fn will be converted to their dagster types before being passed in. For example, if this value is true, enum config passed to config_fn will be actual enums, while if false, then enum config passed to config_fn will be strings.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.configured'>@dagster.configured <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/configurable.py#L253' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.configured" class="hash-link"></a></Link></dt>
<dd>
A decorator that makes it easy to create a function-configured version of an object.
The following definition types can be configured using this function:
- [`GraphDefinition`](graphs.mdx#dagster.GraphDefinition)
- [`ExecutorDefinition`](internals.mdx#dagster.ExecutorDefinition)
- [`LoggerDefinition`](loggers.mdx#dagster.LoggerDefinition)
- [`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition)
- [`OpDefinition`](ops.mdx#dagster.OpDefinition)
Using `configured` may result in config values being displayed in the Dagster UI,
so it is not recommended to use this API with sensitive values, such as
secrets.
If the config that will be supplied to the object is constant, you may alternatively invoke this
and call the result with a dict of config values to be curried. Examples of both strategies
below.
Parameters:
- <strong>configurable</strong> (<em>ConfigurableDefinition</em>) – An object that can be configured.
- <strong>config_schema</strong> ([*ConfigSchema*](#dagster.ConfigSchema)) – The config schema that the inputs to the decorated function must satisfy. Alternatively, annotate the config parameter to the decorated function with a subclass of [`Config`](#dagster.Config) and omit this argument.
- <strong>**kwargs</strong> – Arbitrary keyword arguments that will be passed to the initializer of the returned object.
Returns: (Callable[[Union[Any, Callable[[Any], Any]]], ConfigurableDefinition])
<strong>Examples:</strong>
```python
class GreetingConfig(Config):
message: str
@op
def greeting_op(config: GreetingConfig):
print(config.message)
class HelloConfig(Config):
name: str
@configured(greeting_op)
def hello_op(config: HelloConfig):
return GreetingConfig(message=f"Hello, {config.name}!")
```
```python
dev_s3 = configured(S3Resource, name="dev_s3")({'bucket': 'dev'})
@configured(S3Resource)
def dev_s3(_):
return {'bucket': 'dev'}
@configured(S3Resource, {'bucket_prefix', str})
def dev_s3(config):
return {'bucket': config['bucket_prefix'] + 'dev'}
```
</dd>
</dl>
</div></div>
---
---
title: 'definitions'
sidebar_position: 1000
title_meta: 'definitions API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'definitions Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="definitions">
# Definitions
<dl>
<dt><Link class="anchor" id='dagster.Definitions'>`class` dagster.Definitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L325' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions" class="hash-link"></a></Link></dt>
<dd>
A set of definitions explicitly available and loadable by Dagster tools.
Parameters:
- <strong>assets</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>, </em><em>CacheableAssetsDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of assets. Assets can be created by annotating a function with [`@asset`](assets.mdx#dagster.asset) or [`@observable_source_asset`](assets.mdx#dagster.observable_source_asset). Or they can by directly instantiating [`AssetsDefinition`](assets.mdx#dagster.AssetsDefinition), [`SourceAsset`](assets.mdx#dagster.SourceAsset), or `CacheableAssetsDefinition`.
- <strong>asset_checks</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em>[*AssetChecksDefinition*](asset-checks.mdx#dagster.AssetChecksDefinition)<em>]</em><em>]</em>) – A list of asset checks.
- <strong>schedules</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em><em>Union</em><em>[</em>[*ScheduleDefinition*](schedules-sensors.mdx#dagster.ScheduleDefinition)<em>, </em><em>UnresolvedPartitionedAssetScheduleDefinition</em><em>]</em><em>]</em><em>]</em>) – List of schedules.
- <strong>sensors</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em>[*SensorDefinition*](schedules-sensors.mdx#dagster.SensorDefinition)<em>]</em><em>]</em>) – List of sensors, typically created with [`@sensor`](schedules-sensors.mdx#dagster.sensor).
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Iterable</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – List of jobs. Typically created with [`define_asset_job`](assets.mdx#dagster.define_asset_job) or with [`@job`](jobs.mdx#dagster.job) for jobs defined in terms of ops directly. Jobs created with [`@job`](jobs.mdx#dagster.job) must already have resources bound at job creation time. They do not respect the <cite>resources</cite> argument here.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets. The resources dictionary takes raw Python objects, not just instances of [`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition). If that raw object inherits from [`IOManager`](io-managers.mdx#dagster.IOManager), it gets coerced to an [`IOManagerDefinition`](io-managers.mdx#dagster.IOManagerDefinition). Any other object is coerced to a [`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition). These resources will be automatically bound to any assets passed to this Definitions instance using [`with_resources`](resources.mdx#dagster.with_resources). Assets passed to Definitions with resources already bound using [`with_resources`](resources.mdx#dagster.with_resources) will override this dictionary.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>, </em>[*Executor*](internals.mdx#dagster.Executor)<em>]</em><em>]</em>) – Default executor for jobs. Individual jobs can override this and define their own executors by setting the executor on [`@job`](jobs.mdx#dagster.job) or [`define_asset_job`](assets.mdx#dagster.define_asset_job) explicitly. This executor will also be used for materializing assets directly outside of the context of jobs. If an [`Executor`](internals.mdx#dagster.Executor) is passed, it is coerced into an [`ExecutorDefinition`](internals.mdx#dagster.ExecutorDefinition).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em>) – Default loggers for jobs. Individual jobs can define their own loggers by setting them explictly.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>MetadataMapping</em><em>]</em>) – Arbitrary metadata for the Definitions. Not displayed in the UI but accessible on the Definitions instance at runtime.
- <strong>component_tree</strong> (<em>Optional</em><em>[</em>[*ComponentTree*](components.mdx#dagster.ComponentTree)<em>]</em>) – Information about the Components that were used to construct part of this Definitions object.
Example usage:
```python
Definitions(
assets=[asset_one, asset_two],
schedules=[a_schedule],
sensors=[a_sensor],
jobs=[a_job],
resources={
"a_resource": some_resource,
},
asset_checks=[asset_one_check_one]
)
```
Dagster separates user-defined code from system tools such the web server and
the daemon. Rather than loading code directly into process, a tool such as the
webserver interacts with user-defined code over a serialization boundary.
These tools must be able to locate and load this code when they start. Via CLI
arguments or config, they specify a Python module to inspect.
A Python module is loadable by Dagster tools if there is a top-level variable
that is an instance of [`Definitions`](#dagster.Definitions).
<dl>
<dt><Link class="anchor" id='dagster.Definitions.merge'>`static` merge <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L750' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.merge" class="hash-link"></a></Link></dt>
<dd>
Merges multiple Definitions objects into a single Definitions object.
The returned Definitions object has the union of all the definitions in the input
Definitions objects.
Raises an error if the Definitions objects to be merged contain conflicting values for the
same resource key or logger key, or if they have different executors defined.
Examples:
```python
import submodule1
import submodule2
defs = Definitions.merge(submodule1.defs, submodule2.defs)
```
Returns: The merged definitions.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.validate_loadable'>`static` validate_loadable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L718' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.validate_loadable" class="hash-link"></a></Link></dt>
<dd>
Validates that the enclosed definitions will be loadable by Dagster:
- No assets have conflicting keys.
- No jobs, sensors, or schedules have conflicting names.
- All asset jobs can be resolved.
- All resource requirements are satisfied.
- All partition mappings are valid.
Meant to be used in unit tests.
Raises an error if any of the above are not true.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.get_asset_value_loader'>get_asset_value_loader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L614' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.get_asset_value_loader" class="hash-link"></a></Link></dt>
<dd>
Returns an object that can load the contents of assets as Python objects.
Invokes <cite>load_input</cite> on the [`IOManager`](io-managers.mdx#dagster.IOManager) associated with the assets. Avoids
spinning up resources separately for each asset.
Usage:
```python
with defs.get_asset_value_loader() as loader:
asset1 = loader.load_asset_value("asset1")
asset2 = loader.load_asset_value("asset2")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.get_job_def'>get_job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L460' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.get_job_def" class="hash-link"></a></Link></dt>
<dd>
Get a job definition by name. This will only return a <cite>JobDefinition</cite> if it was directly passed in to the <cite>Definitions</cite> object.
If that is not found, the Definitions object is resolved (transforming UnresolvedAssetJobDefinitions to JobDefinitions and an example). It
also finds jobs passed to sensors and schedules and retrieves them from the repository.
After dagster 1.11, this resolution step will not happen, and will throw an error if the job is not found.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.get_schedule_def'>get_schedule_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L550' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.get_schedule_def" class="hash-link"></a></Link></dt>
<dd>
Get a [`ScheduleDefinition`](schedules-sensors.mdx#dagster.ScheduleDefinition) by name.
If your passed-in schedule had resource dependencies, or the job targeted by the schedule had
resource dependencies, those resource dependencies will be fully resolved on the returned object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.get_sensor_def'>get_sensor_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L527' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.get_sensor_def" class="hash-link"></a></Link></dt>
<dd>
Get a [`SensorDefinition`](schedules-sensors.mdx#dagster.SensorDefinition) by name.
If your passed-in sensor had resource dependencies, or the job targeted by the sensor had
resource dependencies, those resource dependencies will be fully resolved on the returned object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.load_asset_value'>load_asset_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L577' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.load_asset_value" class="hash-link"></a></Link></dt>
<dd>
Load the contents of an asset as a Python object.
Invokes <cite>load_input</cite> on the [`IOManager`](io-managers.mdx#dagster.IOManager) associated with the asset.
If you want to load the values of multiple assets, it’s more efficient to use
[`get_asset_value_loader()`](#dagster.Definitions.get_asset_value_loader), which avoids spinning up
resources separately for each asset.
Parameters:
- <strong>asset_key</strong> (<em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em>) – The key of the asset to load.
- <strong>python_type</strong> (<em>Optional</em><em>[</em><em>Type</em><em>]</em>) – The python type to load the asset as. This is what will be returned inside <cite>load_input</cite> by <cite>context.dagster_type.typing_type</cite>.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The partition of the asset to load.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Input metadata to pass to the [`IOManager`](io-managers.mdx#dagster.IOManager) (is equivalent to setting the metadata argument in <cite>In</cite> or <cite>AssetIn</cite>).
Returns: The contents of an asset as a Python object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.map_asset_specs'>map_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L884' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.map_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Map a function over the included AssetSpecs or AssetsDefinitions in this Definitions object, replacing specs in the sequence
or specs in an AssetsDefinitions with the result of the function.
Parameters:
- <strong>func</strong> (<em>Callable</em><em>[</em><em>[</em>[*AssetSpec*](assets.mdx#dagster.AssetSpec)<em>]</em><em>, </em>[*AssetSpec*](assets.mdx#dagster.AssetSpec)<em>]</em>) – The function to apply to each AssetSpec.
- <strong>selection</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em><em>]</em>) – An asset selection to narrow down the set of assets to apply the function to. If not provided, applies to all assets.
Returns: A Definitions object where the AssetSpecs have been replaced with the result of the function where the selection applies.Return type: [Definitions](#dagster.Definitions)
Examples:
```python
import dagster as dg
my_spec = dg.AssetSpec("asset1")
@dg.asset
def asset1(_): ...
@dg.asset
def asset2(_): ...
defs = Definitions(
assets=[asset1, asset2]
)
# Applies to asset1 and asset2
mapped_defs = defs.map_asset_specs(
func=lambda s: s.merge_attributes(metadata={"new_key": "new_value"}),
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.map_resolved_asset_specs'>map_resolved_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L932' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.map_resolved_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Map a function over the included AssetSpecs or AssetsDefinitions in this Definitions object, replacing specs in the sequence.
See map_asset_specs for more details.
Supports selection and therefore requires resolving the Definitions object to a RepositoryDefinition when there is a selection.
Examples:
```python
import dagster as dg
my_spec = dg.AssetSpec("asset1")
@dg.asset
def asset1(_): ...
@dg.asset
def asset2(_): ...
# Applies only to asset1
mapped_defs = defs.map_resolved_asset_specs(
func=lambda s: s.replace_attributes(metadata={"new_key": "new_value"}),
selection="asset1",
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.resolve_all_asset_keys'>resolve_all_asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L850' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.resolve_all_asset_keys" class="hash-link"></a></Link></dt>
<dd>
Returns an AssetKey object for every asset contained inside the resolved Definitions object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Definitions.resolve_all_asset_specs'>resolve_all_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L844' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Definitions.resolve_all_asset_specs" class="hash-link"></a></Link></dt>
<dd>
Returns an AssetSpec object for every asset contained inside the resolved Definitions object.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.definitions'>@dagster.definitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/components/definitions.py#L49' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.definitions" class="hash-link"></a></Link></dt>
<dd>
Decorator that marks a function as an entry point for loading Dagster definitions.
This decorator provides a lazy loading mechanism for Definitions objects, which is the
preferred approach over directly instantiating Definitions at module import time. It
enables Dagster’s tools to discover and load definitions on-demand without executing
the definition creation logic during module imports. The user can also import this
function and import it for test cases.
The decorated function must return a Definitions object and can optionally accept a
ComponentLoadContext parameter, populated when loaded in the context of
autoloaded defs folders in the dg project layout.
Parameters: <strong>fn</strong> – A function that returns a Definitions object. The function can either:
- Accept no parameters: `() -> Definitions`
- Accept a ComponentLoadContext: `(ComponentLoadContext) -> Definitions`Returns: A callable that will invoke the original function and return its
Definitions object when called by Dagster’s loading mechanisms or directly
by the user.Raises: [DagsterInvariantViolationError](errors.mdx#dagster.DagsterInvariantViolationError)<strong>DagsterInvariantViolationError</strong> – If the function signature doesn’t match the expected
patterns (no parameters or exactly one ComponentLoadContext parameter).
Examples:
Basic usage without context:
```python
import dagster as dg
@dg.definitions
def my_definitions():
@dg.asset
def sales_data():
return [1, 2, 3]
return dg.Definitions(assets=[sales_data])
```
Usage with ComponentLoadContext for autoloaded definitions:
```python
import dagster as dg
@dg.definitions
def my_definitions(context: dg.ComponentLoadContext):
@dg.asset
def sales_data():
# Can use context for environment-specific logic
return load_data_from(context.path)
return dg.Definitions(assets=[sales_data])
```
The decorated function can be imported and used by Dagster tools:
```python
# my_definitions.py
@dg.definitions
def defs():
return dg.Definitions(assets=[my_asset])
# dg dev -f my_definitions.py
```
Note: When used in autoloaded defs folders, the ComponentLoadContext provides access to
environment variables and other contextual information for dynamic definition loading.
See also: - [`dagster.Definitions`](#dagster.Definitions): The object that should be returned by the decorated function
- [`dagster.ComponentLoadContext`](components.mdx#dagster.ComponentLoadContext): Context object for autoloaded definitions
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.create_repository_using_definitions_args'>dagster.create_repository_using_definitions_args <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/definitions_class.py#L72' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.create_repository_using_definitions_args" class="hash-link"></a></Link></dt>
<dd>
Create a named repository using the same arguments as [`Definitions`](#dagster.Definitions). In older
versions of Dagster, repositories were the mechanism for organizing assets, schedules, sensors,
and jobs. There could be many repositories per code location. This was a complicated ontology but
gave users a way to organize code locations that contained large numbers of heterogenous definitions.
As a stopgap for those who both want to 1) use the new [`Definitions`](#dagster.Definitions) API and 2) but still
want multiple logical groups of assets in the same code location, we have introduced this function.
Example usage:
```python
named_repo = create_repository_using_definitions_args(
name="a_repo",
assets=[asset_one, asset_two],
schedules=[a_schedule],
sensors=[a_sensor],
jobs=[a_job],
resources={
"a_resource": some_resource,
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_definitions_from_current_module'>dagster.load_definitions_from_current_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_defs_from_module.py#L82' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_definitions_from_current_module" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Constructs the [`dagster.Definitions`](#dagster.Definitions) from the module where this function is called.
Automatically discovers all objects defined at module scope that can be passed into the
[`dagster.Definitions`](#dagster.Definitions) constructor.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets in the loaded [`dagster.Definitions`](#dagster.Definitions).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Default loggers for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own loggers by setting them explicitly.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Executor*](internals.mdx#dagster.Executor)<em>, </em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em><em>]</em>) – Default executor for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own executors by setting them explicitly.
Returns: The [`dagster.Definitions`](#dagster.Definitions) defined in the current module.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_definitions_from_module'>dagster.load_definitions_from_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_defs_from_module.py#L51' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_definitions_from_module" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Constructs the [`dagster.Definitions`](#dagster.Definitions) from the given module. Automatically
discovers all objects defined at module scope that can be passed into the [`dagster.Definitions`](#dagster.Definitions)
constructor.
Parameters:
- <strong>module</strong> (<em>ModuleType</em>) – The Python module to look for [`dagster.Definitions`](#dagster.Definitions) inside.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets in the loaded [`dagster.Definitions`](#dagster.Definitions).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Default loggers for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own loggers by setting them explicitly.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Executor*](internals.mdx#dagster.Executor)<em>, </em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em><em>]</em>) – Default executor for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own executors by setting them explicitly.
Returns: The [`dagster.Definitions`](#dagster.Definitions) defined in the given module.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_definitions_from_modules'>dagster.load_definitions_from_modules <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_defs_from_module.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_definitions_from_modules" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Constructs the [`dagster.Definitions`](#dagster.Definitions) from the given modules. Automatically
discovers all objects defined at module scope that can be passed into the [`dagster.Definitions`](#dagster.Definitions)
constructor.
Parameters:
- <strong>modules</strong> (<em>Iterable</em><em>[</em><em>ModuleType</em><em>]</em>) – The Python modules to look for [`dagster.Definitions`](#dagster.Definitions) inside.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets in the loaded [`dagster.Definitions`](#dagster.Definitions).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Default loggers for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own loggers by setting them explicitly.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Executor*](internals.mdx#dagster.Executor)<em>, </em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em><em>]</em>) – Default executor for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own executors by setting them explicitly.
Returns: The [`dagster.Definitions`](#dagster.Definitions) defined in the given modules.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_definitions_from_package_module'>dagster.load_definitions_from_package_module <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_defs_from_module.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_definitions_from_package_module" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Constructs the [`dagster.Definitions`](#dagster.Definitions) from the given package module. Automatically
discovers all objects defined at module scope that can be passed into the
[`dagster.Definitions`](#dagster.Definitions) constructor.
Parameters:
- <strong>package_module</strong> (<em>ModuleType</em>) – The package module to look for [`dagster.Definitions`](#dagster.Definitions) inside.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets in the loaded [`dagster.Definitions`](#dagster.Definitions).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Default loggers for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own loggers by setting them explicitly.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Executor*](internals.mdx#dagster.Executor)<em>, </em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em><em>]</em>) – Default executor for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own executors by setting them explicitly.
Returns: The [`dagster.Definitions`](#dagster.Definitions) defined in the given package module.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.load_definitions_from_package_name'>dagster.load_definitions_from_package_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/module_loaders/load_defs_from_module.py#L151' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.load_definitions_from_package_name" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Constructs the [`dagster.Definitions`](#dagster.Definitions) from the package module for the given package name.
Automatically discovers all objects defined at module scope that can be passed into the
[`dagster.Definitions`](#dagster.Definitions) constructor.
Parameters:
- <strong>package_name</strong> (<em>str</em>) – The name of the package module to look for [`dagster.Definitions`](#dagster.Definitions) inside.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Dictionary of resources to bind to assets in the loaded [`dagster.Definitions`](#dagster.Definitions).
- <strong>loggers</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Default loggers for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own loggers by setting them explicitly.
- <strong>executor</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Executor*](internals.mdx#dagster.Executor)<em>, </em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em><em>]</em>) – Default executor for jobs in the loaded [`dagster.Definitions`](#dagster.Definitions). Individual jobs can define their own executors by setting them explicitly.
Returns: The [`dagster.Definitions`](#dagster.Definitions) defined in the package module for the given package name.Return type: [Definitions](#dagster.Definitions)
</dd>
</dl>
</div>
---
---
title: 'dynamic mapping & collect'
sidebar_position: 1000
title_meta: 'dynamic mapping & collect API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dynamic mapping & collect Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dynamic-mapping-collect">
# Dynamic mapping & collect
These APIs provide the means for a simple kind of <em>dynamic orchestration</em> — where the work to be orchestrated is determined not at job definition time but at runtime, dependent on data that’s observed as part of job execution.
<dl>
<dt><Link class="anchor" id='dagster.DynamicOut'>`class` dagster.DynamicOut <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/output.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicOut" class="hash-link"></a></Link></dt>
<dd>
Variant of [`Out`](ops.mdx#dagster.Out) for an output that will dynamically alter the graph at
runtime.
When using in a composition function such as [`@graph`](graphs.mdx#dagster.graph),
dynamic outputs must be used with either
- `map` - clone downstream ops for each separate [`DynamicOut`](#dagster.DynamicOut)
- `collect` - gather across all [`DynamicOut`](#dagster.DynamicOut) in to a list
Uses the same constructor as [`Out`](ops.mdx#dagster.Out)
>
```python
@op(
config_schema={
"path": Field(str, default_value=file_relative_path(__file__, "sample"))
},
out=DynamicOut(str),
)
def files_in_directory(context):
path = context.op_config["path"]
dirname, _, filenames = next(os.walk(path))
for file in filenames:
yield DynamicOutput(os.path.join(dirname, file), mapping_key=_clean(file))
@job
def process_directory():
files = files_in_directory()
# use map to invoke an op on each dynamic output
file_results = files.map(process_file)
# use collect to gather the results in to a list
summarize_directory(file_results.collect())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DynamicOutput'>`class` dagster.DynamicOutput <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L185' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicOutput" class="hash-link"></a></Link></dt>
<dd>
Variant of [`Output`](ops.mdx#dagster.Output) used to support
dynamic mapping & collect. Each `DynamicOutput` produced by an op represents
one item in a set that can be processed individually with `map` or gathered
with `collect`.
Each `DynamicOutput` must have a unique `mapping_key` to distinguish it with it’s set.
Parameters:
- <strong>value</strong> (<em>Any</em>) – The value returned by the compute function.
- <strong>mapping_key</strong> (<em>str</em>) – The key that uniquely identifies this dynamic value relative to its peers. This key will be used to identify the downstream ops when mapped, ie `mapped_op[example_mapping_key]`
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the corresponding [`DynamicOut`](#dagster.DynamicOut) defined on the op. (default: “result”)
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>str</em><em>, </em><em>float</em><em>, </em><em>int</em><em>, </em>[*MetadataValue*](metadata.mdx#dagster.MetadataValue)<em>]</em><em>]</em><em>]</em>) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
<dl>
<dt><Link class="anchor" id='dagster.DynamicOutput.mapping_key'>`property` mapping_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L227' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicOutput.mapping_key" class="hash-link"></a></Link></dt>
<dd>
The mapping_key that was set for this DynamicOutput at instantiation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DynamicOutput.output_name'>`property` output_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L239' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicOutput.output_name" class="hash-link"></a></Link></dt>
<dd>
Name of the [`DynamicOut`](#dagster.DynamicOut) defined on the op that this DynamicOut is associated with.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DynamicOutput.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L233' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicOutput.value" class="hash-link"></a></Link></dt>
<dd>
The value that is returned by the compute function for this DynamicOut.
</dd>
</dl>
</dd>
</dl>
</div>
---
---
title: 'errors'
sidebar_position: 1000
title_meta: 'errors API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'errors Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="module-dagster._core.errors"> <Link id="errors"></Link>
# Errors
Core Dagster error classes.
All errors thrown by the Dagster framework inherit from [`DagsterError`](#dagster.DagsterError). Users
should not subclass this base class for their own exceptions.
There is another exception base class, [`DagsterUserCodeExecutionError`](#dagster.DagsterUserCodeExecutionError), which is
used by the framework in concert with the [`user_code_error_boundary()`](internals.mdx#dagster._core.errors.user_code_error_boundary).
Dagster uses this construct to wrap user code into which it calls. User code can perform arbitrary
computations and may itself throw exceptions. The error boundary catches these user code-generated
exceptions, and then reraises them wrapped in a subclass of
[`DagsterUserCodeExecutionError`](#dagster.DagsterUserCodeExecutionError).
The wrapped exceptions include additional context for the original exceptions, injected by the
Dagster runtime.
<dl>
<dt><Link class="anchor" id='dagster.DagsterError'>`exception` dagster.DagsterError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-shared/dagster_shared/error.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterError" class="hash-link"></a></Link></dt>
<dd>
Base class for all errors thrown by the Dagster framework.
Users should not subclass this base class for their own exceptions.
<dl>
<dt><Link class="anchor" id='dagster.DagsterError.is_user_code_error'>`property` is_user_code_error <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-shared/dagster_shared/error.py#L20' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterError.is_user_code_error" class="hash-link"></a></Link></dt>
<dd>
Returns true if this error is attributable to user code.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterConfigMappingFunctionError'>`exception` dagster.DagsterConfigMappingFunctionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L365' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterConfigMappingFunctionError" class="hash-link"></a></Link></dt>
<dd>
Indicates that an unexpected error occurred while executing the body of a config mapping
function defined in a [`JobDefinition`](jobs.mdx#dagster.JobDefinition) or <cite>~dagster.GraphDefinition</cite> during
config parsing.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventLogInvalidForRun'>`exception` dagster.DagsterEventLogInvalidForRun <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L599' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEventLogInvalidForRun" class="hash-link"></a></Link></dt>
<dd>
Raised when the event logs for a historical run are malformed or invalid.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterExecutionStepExecutionError'>`exception` dagster.DagsterExecutionStepExecutionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L349' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterExecutionStepExecutionError" class="hash-link"></a></Link></dt>
<dd>
Indicates an error occurred while executing the body of an execution step.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterExecutionStepNotFoundError'>`exception` dagster.DagsterExecutionStepNotFoundError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L208' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterExecutionStepNotFoundError" class="hash-link"></a></Link></dt>
<dd>
Thrown when the user specifies execution step keys that do not exist.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInvalidConfigError'>`exception` dagster.DagsterInvalidConfigError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L401' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInvalidConfigError" class="hash-link"></a></Link></dt>
<dd>
Thrown when provided config is invalid (does not type check against the relevant config
schema).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInvalidConfigDefinitionError'>`exception` dagster.DagsterInvalidConfigDefinitionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L158' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInvalidConfigDefinitionError" class="hash-link"></a></Link></dt>
<dd>
Indicates that you have attempted to construct a config with an invalid value.
Acceptable values for config types are any of:
1. A Python primitive type that resolves to a Dagster config type
(`python:int`, `python:float`, `python:bool`, `python:str`, or `python:list`).
2. A Dagster config type: `Int`, `Float`,
`Bool`, `String`, [`StringSource`](config.mdx#dagster.StringSource), `Any`, [`Array`](config.mdx#dagster.Array), [`Noneable`](config.mdx#dagster.Noneable), [`Enum`](config.mdx#dagster.Enum), [`Selector`](config.mdx#dagster.Selector), [`Shape`](config.mdx#dagster.Shape), or [`Permissive`](config.mdx#dagster.Permissive).
3. A bare python dictionary, which will be automatically wrapped in
[`Shape`](config.mdx#dagster.Shape). Values of the dictionary are resolved recursively according to the same rules.
4. A bare python list of length one which itself is config type.
Becomes [`Array`](config.mdx#dagster.Array) with list element as an argument.
5. An instance of [`Field`](config.mdx#dagster.Field).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInvalidDefinitionError'>`exception` dagster.DagsterInvalidDefinitionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L41' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInvalidDefinitionError" class="hash-link"></a></Link></dt>
<dd>
Indicates that the rules for a definition have been violated by the user.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInvalidSubsetError'>`exception` dagster.DagsterInvalidSubsetError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L49' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInvalidSubsetError" class="hash-link"></a></Link></dt>
<dd>
Indicates that a subset of a pipeline is invalid because either:
- One or more ops in the specified subset do not exist on the job.’
- The subset produces an invalid job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInvariantViolationError'>`exception` dagster.DagsterInvariantViolationError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInvariantViolationError" class="hash-link"></a></Link></dt>
<dd>
Indicates the user has violated a well-defined invariant that can only be enforced
at runtime.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterResourceFunctionError'>`exception` dagster.DagsterResourceFunctionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L359' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterResourceFunctionError" class="hash-link"></a></Link></dt>
<dd>
Indicates an error occurred while executing the body of the `resource_fn` in a
[`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition) during resource initialization.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunNotFoundError'>`exception` dagster.DagsterRunNotFoundError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRunNotFoundError" class="hash-link"></a></Link></dt>
<dd>
Thrown when a run cannot be found in run storage.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterStepOutputNotFoundError'>`exception` dagster.DagsterStepOutputNotFoundError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L228' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterStepOutputNotFoundError" class="hash-link"></a></Link></dt>
<dd>
Indicates that previous step outputs required for an execution step to proceed are not
available.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterSubprocessError'>`exception` dagster.DagsterSubprocessError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L432' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterSubprocessError" class="hash-link"></a></Link></dt>
<dd>
An exception has occurred in one or more of the child processes dagster manages.
This error forwards the message and stack trace for all of the collected errors.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeCheckDidNotPass'>`exception` dagster.DagsterTypeCheckDidNotPass <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L574' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeCheckDidNotPass" class="hash-link"></a></Link></dt>
<dd>
Indicates that a type check failed.
This is raised when `raise_on_error` is `True` in calls to the synchronous job and
graph execution APIs (e.g. <cite>graph.execute_in_process()</cite>, <cite>job.execute_in_process()</cite> – typically
within a test), and a [`DagsterType`](types.mdx#dagster.DagsterType)’s type check fails by returning either
`False` or an instance of [`TypeCheck`](ops.mdx#dagster.TypeCheck) whose `success` member is `False`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeCheckError'>`exception` dagster.DagsterTypeCheckError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L325' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeCheckError" class="hash-link"></a></Link></dt>
<dd>
Indicates an error in the op type system at runtime. E.g. a op receives an
unexpected input, or produces an output that does not match the type of the output definition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterUnknownResourceError'>`exception` dagster.DagsterUnknownResourceError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L378' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterUnknownResourceError" class="hash-link"></a></Link></dt>
<dd>
Indicates that an unknown resource was accessed in the body of an execution step. May often
happen by accessing a resource in the compute function of an op without first supplying the
op with the correct <cite>required_resource_keys</cite> argument.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterUnmetExecutorRequirementsError'>`exception` dagster.DagsterUnmetExecutorRequirementsError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L426' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterUnmetExecutorRequirementsError" class="hash-link"></a></Link></dt>
<dd>
Indicates the resolved executor is incompatible with the state of other systems
such as the `DagsterInstance` or system storage configuration.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterUserCodeExecutionError'>`exception` dagster.DagsterUserCodeExecutionError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L295' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterUserCodeExecutionError" class="hash-link"></a></Link></dt>
<dd>
This is the base class for any exception that is meant to wrap an
`python:Exception` thrown by user code. It wraps that existing user code.
The `original_exc_info` argument to the constructor is meant to be a tuple of the type
returned by `sys.exc_info` at the call site of the constructor.
Users should not subclass this base class for their own exceptions and should instead throw
freely from user code. User exceptions will be automatically wrapped and rethrown.
<dl>
<dt><Link class="anchor" id='dagster.DagsterUserCodeExecutionError.is_user_code_error'>`property` is_user_code_error <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L320' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterUserCodeExecutionError.is_user_code_error" class="hash-link"></a></Link></dt>
<dd>
Returns true if this error is attributable to user code.
</dd>
</dl>
</dd>
</dl>
</div>
---
---
title: 'execution'
sidebar_position: 1000
title_meta: 'execution API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'execution Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="execution">
# Execution
<div class="section" id="materializing-assets">
## Materializing assets
<dl>
<dt><Link class="anchor" id='dagster.materialize'>dagster.materialize <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/materialize.py#L24' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.materialize" class="hash-link"></a></Link></dt>
<dd>
Executes a single-threaded, in-process run which materializes provided assets.
By default, will materialize assets to the local filesystem.
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*AssetSpec*](assets.mdx#dagster.AssetSpec)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em>) –
The assets to materialize.
Unless you’re using <cite>deps</cite> or <cite>non_argument_deps</cite>, you must also include all assets that are upstream of the assets that you want to materialize. This is because those upstream asset definitions have information that is needed to load their contents while materializing the downstream assets.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – The resources needed for execution. Can provide resource instances directly, or resource definitions. Note that if provided resources conflict with resources directly on assets, an error will be thrown.
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The run config to use for the run that materializes the assets.
- <strong>partition_key</strong> – (Optional[str]) The string partition key that specifies the run config to execute. Can only be used to select run config for assets with partitioned config.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for the run.
- <strong>selection</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em><em>]</em>) –
A sub-selection of assets to materialize.
If not provided, then all assets will be materialized.
Returns: The result of the execution.Return type: [ExecuteInProcessResult](#dagster.ExecuteInProcessResult)
Examples:
```python
@asset
def asset1():
...
@asset
def asset2(asset1):
...
# executes a run that materializes asset1 and then asset2
materialize([asset1, asset2])
# executes a run that materializes just asset2, loading its input from asset1
materialize([asset1, asset2], selection=[asset2])
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.materialize_to_memory'>dagster.materialize_to_memory <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/materialize.py#L118' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.materialize_to_memory" class="hash-link"></a></Link></dt>
<dd>
Executes a single-threaded, in-process run which materializes provided assets in memory.
Will explicitly use [`mem_io_manager()`](io-managers.mdx#dagster.mem_io_manager) for all required io manager
keys. If any io managers are directly provided using the <cite>resources</cite>
argument, a [`DagsterInvariantViolationError`](errors.mdx#dagster.DagsterInvariantViolationError) will be thrown.
Parameters:
- <strong>assets</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*AssetSpec*](assets.mdx#dagster.AssetSpec)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em>) – The assets to materialize. Can also provide [`SourceAsset`](assets.mdx#dagster.SourceAsset) objects to fill dependencies for asset defs.
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The run config to use for the run that materializes the assets.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – The resources needed for execution. Can provide resource instances directly, or resource definitions. If provided resources conflict with resources directly on assets, an error will be thrown.
- <strong>partition_key</strong> – (Optional[str]) The string partition key that specifies the run config to execute. Can only be used to select run config for assets with partitioned config.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags for the run.
- <strong>selection</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em><em>]</em>) –
A sub-selection of assets to materialize.
If not provided, then all assets will be materialized.
Returns: The result of the execution.Return type: [ExecuteInProcessResult](#dagster.ExecuteInProcessResult)
Examples:
```python
@asset
def asset1():
...
@asset
def asset2(asset1):
...
# executes a run that materializes asset1 and then asset2
materialize([asset1, asset2])
# executes a run that materializes just asset1
materialize([asset1, asset2], selection=[asset1])
```
</dd>
</dl>
</div>
<div class="section" id="executing-jobs">
## Executing jobs
<dl>
<dt>`class` dagster.JobDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L726' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Defines a Dagster job.
<dl>
<dt>execute_in_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L732' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Execute the Job in-process, gathering results in-memory.
The <cite>executor_def</cite> on the Job will be ignored, and replaced with the in-process executor.
If using the default <cite>io_manager</cite>, it will switch from filesystem to in-memory.
Parameters:
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The configuration for the run
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The instance to execute against, an ephemeral one will be used if none provided.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The string partition key that specifies the run config to execute. Can only be used to select run config for jobs with partitioned config.
- <strong>raise_on_error</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to raise exceptions when they occur. Defaults to `True`.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of op selection queries (including single op names) to execute. For example: * `['some_op']`: selects `some_op` itself. * `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies). * `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down. * `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of the job. Input values provided here will override input values that have been provided to the job directly.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources needed if any are required. Can provide resource instances directly, or resource definitions.
Returns: [`ExecuteInProcessResult`](#dagster.ExecuteInProcessResult)
</dd>
</dl>
<dl>
<dt>run_request_for_partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1052' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.0.
Directly instantiate `RunRequest(partition_key=...)` instead..
:::
Creates a RunRequest object for a run that processes the given partition.
Parameters:
- <strong>partition_key</strong> – The key of the partition to request a run for.
- <strong>run_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string key to identify this launched run. For sensors, ensures that only one run is created per run key across all sensor evaluations. For schedules, ensures that one run is created per tick, across failure recoveries. Passing in a <cite>None</cite> value means that a run will always be launched per evaluation.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A dictionary of tags (string key-value pairs) to attach to the launched run.
- <strong>(</strong><strong>Optional</strong><strong>[</strong><strong>Mapping</strong><strong>[</strong><strong>str</strong> (<em>run_config</em>) – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>Any</strong><strong>]</strong><strong>]</strong> – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – Used to determine which time-partitions exist. Defaults to now.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Required when the partitions definition is a DynamicPartitionsDefinition with a name defined. Users can pass the DagsterInstance fetched via <cite>context.instance</cite> to this argument.
Returns: an object that requests a run to process the given partition.Return type: [RunRequest](schedules-sensors.mdx#dagster.RunRequest)
</dd>
</dl>
<dl>
<dt>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1207' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Apply a set of hooks to all op instances within the job.
</dd>
</dl>
<dl>
<dt>with_top_level_resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1199' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Apply a set of resources to all op instances within the job.
</dd>
</dl>
<dl>
<dt>`property` config_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The config mapping for the job, if it has one.
A config mapping defines a way to map a top-level config schema to run config for the job.
</dd>
</dl>
<dl>
<dt>`property` executor_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L402' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns the default [`ExecutorDefinition`](internals.mdx#dagster.ExecutorDefinition) for the job.
If the user has not specified an executor definition, then this will default to the
[`multi_or_in_process_executor()`](#dagster.multi_or_in_process_executor). If a default is specified on the
[`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to, then that will be used instead.
</dd>
</dl>
<dl>
<dt>`property` has_specified_executor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L413' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns True if this job has explicitly specified an executor, and False if the executor
was inherited through defaults or the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to.
</dd>
</dl>
<dl>
<dt>`property` has_specified_loggers <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L463' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns true if the job explicitly set loggers, and False if loggers were inherited
through defaults or the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to.
</dd>
</dl>
<dl>
<dt>`property` loggers <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns the set of LoggerDefinition objects specified on the job.
If the user has not specified a mapping of [`LoggerDefinition`](loggers.mdx#dagster.LoggerDefinition) objects, then this
will default to the `colored_console_logger()` under the key <cite>console</cite>. If a default
is specified on the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to, then that will
be used instead.
</dd>
</dl>
<dl>
<dt>`property` partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The partitioned config for the job, if it has one.
A partitioned config defines a way to map partition keys to run config for the job.
</dd>
</dl>
<dl>
<dt>`property` partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L485' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns the [`PartitionsDefinition`](partitions.mdx#dagster.PartitionsDefinition) for the job, if it has one.
A partitions definition defines the set of partition keys the job operates on.
</dd>
</dl>
<dl>
<dt>`property` resource_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L421' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Returns the set of ResourceDefinition objects specified on the job.
This may not be the complete set of resources required by the job, since those can also be
provided on the [`Definitions`](definitions.mdx#dagster.Definitions) object the job may be provided to.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.execute_job'>dagster.execute_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/api.py#L319' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.execute_job" class="hash-link"></a></Link></dt>
<dd>
Execute a job synchronously.
This API represents dagster’s python entrypoint for out-of-process
execution. For most testing purposes, `
execute_in_process()` will be more suitable, but when wanting to run
execution using an out-of-process executor (such as `dagster.
multiprocess_executor`), then <cite>execute_job</cite> is suitable.
<cite>execute_job</cite> expects a persistent [`DagsterInstance`](internals.mdx#dagster.DagsterInstance) for
execution, meaning the <cite>$DAGSTER_HOME</cite> environment variable must be set.
It also expects a reconstructable pointer to a [`JobDefinition`](jobs.mdx#dagster.JobDefinition) so
that it can be reconstructed in separate processes. This can be done by
wrapping the `JobDefinition` in a call to `dagster.
reconstructable()`.
```python
from dagster import DagsterInstance, execute_job, job, reconstructable
@job
def the_job():
...
instance = DagsterInstance.get()
result = execute_job(reconstructable(the_job), instance=instance)
assert result.success
```
If using the [`to_job()`](graphs.mdx#dagster.GraphDefinition.to_job) method to
construct the `JobDefinition`, then the invocation must be wrapped in a
module-scope function, which can be passed to `reconstructable`.
```python
from dagster import graph, reconstructable
@graph
def the_graph():
...
def define_job():
return the_graph.to_job(...)
result = execute_job(reconstructable(define_job), ...)
```
Since <cite>execute_job</cite> is potentially executing outside of the current
process, output objects need to be retrieved by use of the provided job’s
io managers. Output objects can be retrieved by opening the result of
<cite>execute_job</cite> as a context manager.
```python
from dagster import execute_job
with execute_job(...) as result:
output_obj = result.output_for_node("some_op")
```
`execute_job` can also be used to reexecute a run, by providing a [`ReexecutionOptions`](#dagster.ReexecutionOptions) object.
```python
from dagster import ReexecutionOptions, execute_job
instance = DagsterInstance.get()
options = ReexecutionOptions.from_failure(run_id=failed_run_id, instance=instance)
execute_job(reconstructable(job), instance=instance, reexecution_options=options)
```
Parameters:
- <strong>job</strong> (<em>ReconstructableJob</em>) – A reconstructable pointer to a [`JobDefinition`](jobs.mdx#dagster.JobDefinition).
- <strong>instance</strong> ([*DagsterInstance*](internals.mdx#dagster.DagsterInstance)) – The instance to execute against.
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) – The configuration that parametrizes this run, as a dict.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Arbitrary key-value pairs that will be added to run logs.
- <strong>raise_on_error</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to raise exceptions when they occur. Defaults to `False`.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) –
A list of op selection queries (including single op names) to execute. For example:
- `['some_op']`: selects `some_op` itself.
- `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies).
- `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down.
- `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
- <strong>reexecution_options</strong> (<em>Optional</em><em>[</em>[*ReexecutionOptions*](#dagster.ReexecutionOptions)<em>]</em>) – Reexecution options to provide to the run, if this run is intended to be a reexecution of a previous run. Cannot be used in tandem with the `op_selection` argument.
Returns: The result of job execution.Return type: [`JobExecutionResult`](#dagster.JobExecutionResult)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ReexecutionOptions'>`class` dagster.ReexecutionOptions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/api.py#L280' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ReexecutionOptions" class="hash-link"></a></Link></dt>
<dd>
Reexecution options for python-based execution in Dagster.
Parameters:
- <strong>parent_run_id</strong> (<em>str</em>) – The run_id of the run to reexecute.
- <strong>step_selection</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) –
The list of step selections to reexecute. Must be a subset or match of the set of steps executed in the original run. For example:
- `['some_op']`: selects `some_op` itself.
- `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies).
- `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down.
- `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.instance_for_test'>dagster.instance_for_test <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance_for_test.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.instance_for_test" class="hash-link"></a></Link></dt>
<dd>
Creates a persistent [`DagsterInstance`](internals.mdx#dagster.DagsterInstance) available within a context manager.
When a context manager is opened, if no <cite>temp_dir</cite> parameter is set, a new
temporary directory will be created for the duration of the context
manager’s opening. If the <cite>set_dagster_home</cite> parameter is set to True
(True by default), the <cite>$DAGSTER_HOME</cite> environment variable will be
overridden to be this directory (or the directory passed in by <cite>temp_dir</cite>)
for the duration of the context manager being open.
Parameters:
- <strong>overrides</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Config to provide to instance (config format follows that typically found in an <cite>instance.yaml</cite> file).
- <strong>set_dagster_home</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If set to True, the <cite>$DAGSTER_HOME</cite> environment variable will be overridden to be the directory used by this instance for the duration that the context manager is open. Upon the context manager closing, the <cite>$DAGSTER_HOME</cite> variable will be re-set to the original value. (Defaults to True).
- <strong>temp_dir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The directory to use for storing local artifacts produced by the instance. If not set, a temporary directory will be created for the duration of the context manager being open, and all artifacts will be torn down afterward.
</dd>
</dl>
</div>
<div class="section" id="executing-graphs">
## Executing graphs
<dl>
<dt>`class` dagster.GraphDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/node_definition.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Defines a Dagster op graph.
An op graph is made up of
- Nodes, which can either be an op (the functional unit of computation), or another graph.
- Dependencies, which determine how the values produced by nodes as outputs flow from one node to another. This tells Dagster how to arrange nodes into a directed, acyclic graph (DAG) of compute.
End users should prefer the [`@graph`](graphs.mdx#dagster.graph) decorator. GraphDefinition is generally
intended to be used by framework authors or for programatically generated graphs.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the graph. Must be unique within any [`GraphDefinition`](graphs.mdx#dagster.GraphDefinition) or [`JobDefinition`](jobs.mdx#dagster.JobDefinition) containing the graph.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the job.
- <strong>node_defs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>NodeDefinition</em><em>]</em><em>]</em>) – The set of ops / graphs used in this graph.
- <strong>dependencies</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em>[*NodeInvocation*](graphs.mdx#dagster.NodeInvocation)<em>]</em><em>, </em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*DependencyDefinition*](graphs.mdx#dagster.DependencyDefinition)<em>]</em><em>]</em><em>]</em>) – A structure that declares the dependencies of each op’s inputs on the outputs of other ops in the graph. Keys of the top level dict are either the string names of ops in the graph or, in the case of aliased ops, [`NodeInvocations`](graphs.mdx#dagster.NodeInvocation). Values of the top level dict are themselves dicts, which map input names belonging to the op or aliased op to [`DependencyDefinitions`](graphs.mdx#dagster.DependencyDefinition).
- <strong>input_mappings</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*InputMapping*](graphs.mdx#dagster.InputMapping)<em>]</em><em>]</em>) – Defines the inputs to the nested graph, and how they map to the inputs of its constituent ops.
- <strong>output_mappings</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*OutputMapping*](graphs.mdx#dagster.OutputMapping)<em>]</em><em>]</em>) – Defines the outputs of the nested graph, and how they map from the outputs of its constituent ops.
- <strong>config</strong> (<em>Optional</em><em>[</em>[*ConfigMapping*](config.mdx#dagster.ConfigMapping)<em>]</em>) – Defines the config of the graph, and how its schema maps to the config of its constituent ops.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Arbitrary metadata for any execution of the graph. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>. These tag values may be overwritten by tag values provided at invocation time.
- <strong>composition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>]</em>) – The function that defines this graph. Used to generate code references for this graph.
Examples:
```python
@op
def return_one():
return 1
@op
def add_one(num):
return num + 1
graph_def = GraphDefinition(
name='basic',
node_defs=[return_one, add_one],
dependencies={'add_one': {'num': DependencyDefinition('return_one')}},
)
```
<dl>
<dt>alias <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L828' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Aliases the graph with a new name.
Can only be used in the context of a [`@graph`](graphs.mdx#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.alias("my_graph_alias")
```
</dd>
</dl>
<dl>
<dt>execute_in_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L724' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Execute this graph in-process, collecting results in-memory.
Parameters:
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Run config to provide to execution. The configuration for the underlying graph should exist under the “ops” key.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The instance to execute against, an ephemeral one will be used if none provided.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources needed if any are required. Can provide resource instances directly, or resource definitions.
- <strong>raise_on_error</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to raise exceptions when they occur. Defaults to `True`.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of op selection queries (including single op names) to execute. For example: * `['some_op']`: selects `some_op` itself. * `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies). * `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down. * `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of the graph.
Returns: [`ExecuteInProcessResult`](#dagster.ExecuteInProcessResult)
</dd>
</dl>
<dl>
<dt>tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L843' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Attaches the provided tags to the graph immutably.
Can only be used in the context of a [`@graph`](graphs.mdx#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.tag({"my_tag": "my_value"})
```
</dd>
</dl>
<dl>
<dt>to_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L601' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Make this graph in to an executable Job by providing remaining components required for execution.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name for the Job. Defaults to the name of the this graph.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em> [</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – Resources that are required by this graph for execution. If not defined, <cite>io_manager</cite> will default to filesystem.
- <strong>config</strong> –
Describes how the job is parameterized at runtime.
If no value is provided, then the schema for the job’s run config is a standard format based on its ops and resources.
If a dictionary is provided, then it must conform to the standard config schema, and it will be used as the job’s run config for the job whenever the job is executed. The values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
If a [`ConfigMapping`](config.mdx#dagster.ConfigMapping) object is provided, then the schema for the job’s run config is determined by the config mapping, and the ConfigMapping, which should return configuration in the standard format to configure the job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the job and can be used for searching and filtering in the UI. Values that are not already strings will be serialized as JSON. If <cite>run_tags</cite> is not set, then the content of <cite>tags</cite> will also be automatically appended to the tags of any runs of this job.
- <strong>run_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that will be automatically attached to runs launched by this job. Values that are not already strings will be serialized as JSON. These tag values may be overwritten by tag values provided at invocation time. If <cite>run_tags</cite> is set, then <cite>tags</cite> are not automatically appended to the tags of any runs of this job.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary information that will be attached to the JobDefinition and be viewable in the Dagster UI. Keys must be strings, and values must be python primitive types or one of the provided MetadataValue types
- <strong>logger_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – A dictionary of string logger identifiers to their implementations.
- <strong>executor_def</strong> (<em>Optional</em><em>[</em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em>) – How this Job will be executed. Defaults to [`multi_or_in_process_executor`](#dagster.multi_or_in_process_executor), which can be switched between multi-process and in-process modes of execution. The default mode of execution is multi-process.
- <strong>op_retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The default retry policy for all ops in this job. Only used if retry policy is not defined on the op definition or op invocation.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines a discrete set of partition keys that can parameterize the job. If this argument is supplied, the config argument can’t also be supplied.
- <strong>asset_layer</strong> (<em>Optional</em><em>[</em><em>AssetLayer</em><em>]</em>) – Top level information about the assets this job will produce. Generally should not be set manually.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of a job.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A sequence of strings identifying the owners of the job.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](assets.mdx#dagster.AutomationCondition)<em>]</em>) – An experimental, job-scoped automation condition. Only supported for asset jobs; generally set via <cite>define_asset_job</cite> rather than here.
Returns: JobDefinition
</dd>
</dl>
<dl>
<dt>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L858' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Attaches the provided hooks to the graph immutably.
Can only be used in the context of a [`@graph`](graphs.mdx#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.with_hooks({my_hook})
```
</dd>
</dl>
<dl>
<dt>with_retry_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L873' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Attaches the provided retry policy to the graph immutably.
Can only be used in the context of a [`@graph`](graphs.mdx#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.with_retry_policy(RetryPolicy(max_retries=5))
```
</dd>
</dl>
<dl>
<dt>`property` config_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The config mapping for the graph, if present.
By specifying a config mapping function, you can override the configuration for the child nodes contained within a graph.
</dd>
</dl>
<dl>
<dt>`property` input_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L413' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Input mappings for the graph.
An input mapping is a mapping from an input of the graph to an input of a child node.
</dd>
</dl>
<dl>
<dt>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L809' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The name of the graph.
</dd>
</dl>
<dl>
<dt>`property` output_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L422' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Output mappings for the graph.
An output mapping is a mapping from an output of the graph to an output of a child node.
</dd>
</dl>
<dl>
<dt>`property` tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L815' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The tags associated with the graph.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="execution-results">
## Execution results
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult'>`class` dagster.ExecuteInProcessResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L16' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult" class="hash-link"></a></Link></dt>
<dd>
Result object returned by in-process testing APIs.
Users should not instantiate this object directly. Used for retrieving run success, events, and outputs from execution methods that return this object.
This object is returned by:
- [`dagster.GraphDefinition.execute_in_process()`](graphs.mdx#dagster.GraphDefinition.execute_in_process)
- [`dagster.JobDefinition.execute_in_process()`](jobs.mdx#dagster.JobDefinition.execute_in_process)
- [`dagster.materialize_to_memory()`](#dagster.materialize_to_memory)
- [`dagster.materialize()`](#dagster.materialize)
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.asset_value'>asset_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L123' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.asset_value" class="hash-link"></a></Link></dt>
<dd>
Retrieves the value of an asset that was materialized during the execution of the job.
Parameters: <strong>asset_key</strong> (<em>CoercibleToAssetKey</em>) – The key of the asset to retrieve.Returns: The value of the retrieved asset.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.output_for_node'>output_for_node <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L108' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.output_for_node" class="hash-link"></a></Link></dt>
<dd>
Retrieves output value with a particular name from the in-process run of the job.
Parameters:
- <strong>node_str</strong> (<em>str</em>) – Name of the op/graph whose output should be retrieved. If the intended graph/op is nested within another graph, the syntax is <cite>outer_graph.inner_node</cite>.
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the output on the op/graph to retrieve. Defaults to <cite>result</cite>, the default output name in dagster.
Returns: The value of the retrieved output.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.output_value'>output_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L140' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.output_value" class="hash-link"></a></Link></dt>
<dd>
Retrieves output of top-level job, if an output is returned.
Parameters: <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output to retrieve. Defaults to <cite>result</cite>,
the default output name in dagster.Returns: The value of the retrieved output.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.all_events'>`property` all_events <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.all_events" class="hash-link"></a></Link></dt>
<dd>
All dagster events emitted during execution.
Type: List[[DagsterEvent](#dagster.DagsterEvent)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.dagster_run'>`property` dagster_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.dagster_run" class="hash-link"></a></Link></dt>
<dd>
The Dagster run that was executed.
Type: [DagsterRun](internals.mdx#dagster.DagsterRun)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L51' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.job_def" class="hash-link"></a></Link></dt>
<dd>
The job definition that was executed.
Type: [JobDefinition](jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecuteInProcessResult.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/execute_in_process_result.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecuteInProcessResult.run_id" class="hash-link"></a></Link></dt>
<dd>
The run ID of the executed [`DagsterRun`](internals.mdx#dagster.DagsterRun).
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult'>`class` dagster.JobExecutionResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult" class="hash-link"></a></Link></dt>
<dd>
Result object returned by [`dagster.execute_job()`](#dagster.execute_job).
Used for retrieving run success, events, and outputs from <cite>execute_job</cite>.
Users should not directly instantiate this class.
Events and run information can be retrieved off of the object directly. In
order to access outputs, the <cite>ExecuteJobResult</cite> object needs to be opened
as a context manager, which will re-initialize the resources from
execution.
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.output_for_node'>output_for_node <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L93' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.output_for_node" class="hash-link"></a></Link></dt>
<dd>
Retrieves output value with a particular name from the run of the job.
In order to use this method, the <cite>ExecuteJobResult</cite> object must be opened as a context manager. If this method is used without opening the context manager, it will result in a [`DagsterInvariantViolationError`](errors.mdx#dagster.DagsterInvariantViolationError).
Parameters:
- <strong>node_str</strong> (<em>str</em>) – Name of the op/graph whose output should be retrieved. If the intended graph/op is nested within another graph, the syntax is <cite>outer_graph.inner_node</cite>.
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the output on the op/graph to retrieve. Defaults to <cite>result</cite>, the default output name in dagster.
Returns: The value of the retrieved output.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.output_value'>output_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.output_value" class="hash-link"></a></Link></dt>
<dd>
Retrieves output of top-level job, if an output is returned.
In order to use this method, the <cite>ExecuteJobResult</cite> object must be opened as a context manager. If this method is used without opening the context manager, it will result in a [`DagsterInvariantViolationError`](errors.mdx#dagster.DagsterInvariantViolationError). If the top-level job has no output, calling this method will also result in a [`DagsterInvariantViolationError`](errors.mdx#dagster.DagsterInvariantViolationError).
Parameters: <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output to retrieve. Defaults to <cite>result</cite>,
the default output name in dagster.Returns: The value of the retrieved output.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.all_events'>`property` all_events <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.all_events" class="hash-link"></a></Link></dt>
<dd>
List of all events yielded by the job execution.
Type: Sequence[[DagsterEvent](#dagster.DagsterEvent)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.dagster_run'>`property` dagster_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L60' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.dagster_run" class="hash-link"></a></Link></dt>
<dd>
The Dagster run that was executed.
Type: [DagsterRun](internals.mdx#dagster.DagsterRun)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.job_def" class="hash-link"></a></Link></dt>
<dd>
The job definition that was executed.
Type: [JobDefinition](jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobExecutionResult.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/job_execution_result.py#L72' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobExecutionResult.run_id" class="hash-link"></a></Link></dt>
<dd>
The id of the Dagster run that was executed.
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent'>`class` dagster.DagsterEvent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L455' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent" class="hash-link"></a></Link></dt>
<dd>
Events yielded by op and job execution.
Users should not instantiate this class.
Parameters:
- <strong>event_type_value</strong> (<em>str</em>) – Value for a DagsterEventType.
- <strong>job_name</strong> (<em>str</em>)
- <strong>node_handle</strong> (<em>NodeHandle</em>)
- <strong>step_kind_value</strong> (<em>str</em>) – Value for a StepKind.
- <strong>logging_tags</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em>)
- <strong>event_specific_data</strong> (<em>Any</em>) – Type must correspond to event_type_value.
- <strong>message</strong> (<em>str</em>)
- <strong>pid</strong> (<em>int</em>)
- <strong>step_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – DEPRECATED
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L757' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.asset_key" class="hash-link"></a></Link></dt>
<dd>
For events that correspond to a specific asset_key / partition
(ASSET_MATERIALIZTION, ASSET_OBSERVATION, ASSET_MATERIALIZATION_PLANNED), returns that
asset key. Otherwise, returns None.
Type: Optional[[AssetKey](assets.mdx#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.event_type'>`property` event_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L612' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.event_type" class="hash-link"></a></Link></dt>
<dd>
The type of this event.
Type: [DagsterEventType](#dagster.DagsterEventType)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_asset_materialization_planned'>`property` is_asset_materialization_planned <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L746' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_asset_materialization_planned" class="hash-link"></a></Link></dt>
<dd>
If this event is of type ASSET_MATERIALIZATION_PLANNED.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_asset_observation'>`property` is_asset_observation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L740' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_asset_observation" class="hash-link"></a></Link></dt>
<dd>
If this event is of type ASSET_OBSERVATION.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_engine_event'>`property` is_engine_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L710' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_engine_event" class="hash-link"></a></Link></dt>
<dd>
If this event is of type ENGINE_EVENT.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_expectation_result'>`property` is_expectation_result <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L734' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_expectation_result" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_EXPECTATION_RESULT.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_failure'>`property` is_failure <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L700' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_failure" class="hash-link"></a></Link></dt>
<dd>
If this event represents the failure of a run or step.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_handled_output'>`property` is_handled_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L716' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_handled_output" class="hash-link"></a></Link></dt>
<dd>
If this event is of type HANDLED_OUTPUT.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_hook_event'>`property` is_hook_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L624' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_hook_event" class="hash-link"></a></Link></dt>
<dd>
If this event relates to the execution of a hook.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_loaded_input'>`property` is_loaded_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L722' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_loaded_input" class="hash-link"></a></Link></dt>
<dd>
If this event is of type LOADED_INPUT.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_resource_init_failure'>`property` is_resource_init_failure <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L664' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_resource_init_failure" class="hash-link"></a></Link></dt>
<dd>
If this event is of type RESOURCE_INIT_FAILURE.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_event'>`property` is_step_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L618' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_event" class="hash-link"></a></Link></dt>
<dd>
If this event relates to a specific step.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_failure'>`property` is_step_failure <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L658' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_failure" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_FAILURE.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_materialization'>`property` is_step_materialization <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L728' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_materialization" class="hash-link"></a></Link></dt>
<dd>
If this event is of type ASSET_MATERIALIZATION.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_restarted'>`property` is_step_restarted <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L682' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_restarted" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_RESTARTED.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_skipped'>`property` is_step_skipped <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L670' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_skipped" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_SKIPPED.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_start'>`property` is_step_start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L652' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_start" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_START.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_success'>`property` is_step_success <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L640' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_success" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_SUCCESS.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_step_up_for_retry'>`property` is_step_up_for_retry <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L676' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_step_up_for_retry" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_UP_FOR_RETRY.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.is_successful_output'>`property` is_successful_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L646' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.is_successful_output" class="hash-link"></a></Link></dt>
<dd>
If this event is of type STEP_OUTPUT.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEvent.partition'>`property` partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L781' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEvent.partition" class="hash-link"></a></Link></dt>
<dd>
For events that correspond to a specific asset_key / partition
(ASSET_MATERIALIZTION, ASSET_OBSERVATION, ASSET_MATERIALIZATION_PLANNED), returns that
partition. Otherwise, returns None.
Type: Optional[[AssetKey](assets.mdx#dagster.AssetKey)]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType'>`class` dagster.DagsterEventType <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/__init__.py#L112' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterEventType" class="hash-link"></a></Link></dt>
<dd>
The types of events that may be yielded by op and job execution.
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ALERT_FAILURE'>ALERT_FAILURE `=` 'ALERT_FAILURE'<a href="#dagster.DagsterEventType.ALERT_FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ALERT_START'>ALERT_START `=` 'ALERT_START'<a href="#dagster.DagsterEventType.ALERT_START" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ALERT_SUCCESS'>ALERT_SUCCESS `=` 'ALERT_SUCCESS'<a href="#dagster.DagsterEventType.ALERT_SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_CHECK_EVALUATION'>ASSET_CHECK_EVALUATION `=` 'ASSET_CHECK_EVALUATION'<a href="#dagster.DagsterEventType.ASSET_CHECK_EVALUATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_CHECK_EVALUATION_PLANNED'>ASSET_CHECK_EVALUATION_PLANNED `=` 'ASSET_CHECK_EVALUATION_PLANNED'<a href="#dagster.DagsterEventType.ASSET_CHECK_EVALUATION_PLANNED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_FAILED_TO_MATERIALIZE'>ASSET_FAILED_TO_MATERIALIZE `=` 'ASSET_FAILED_TO_MATERIALIZE'<a href="#dagster.DagsterEventType.ASSET_FAILED_TO_MATERIALIZE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_HEALTH_CHANGED'>ASSET_HEALTH_CHANGED `=` 'ASSET_HEALTH_CHANGED'<a href="#dagster.DagsterEventType.ASSET_HEALTH_CHANGED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_MATERIALIZATION'>ASSET_MATERIALIZATION `=` 'ASSET_MATERIALIZATION'<a href="#dagster.DagsterEventType.ASSET_MATERIALIZATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_MATERIALIZATION_PLANNED'>ASSET_MATERIALIZATION_PLANNED `=` 'ASSET_MATERIALIZATION_PLANNED'<a href="#dagster.DagsterEventType.ASSET_MATERIALIZATION_PLANNED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_OBSERVATION'>ASSET_OBSERVATION `=` 'ASSET_OBSERVATION'<a href="#dagster.DagsterEventType.ASSET_OBSERVATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_STORE_OPERATION'>ASSET_STORE_OPERATION `=` 'ASSET_STORE_OPERATION'<a href="#dagster.DagsterEventType.ASSET_STORE_OPERATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ASSET_WIPED'>ASSET_WIPED `=` 'ASSET_WIPED'<a href="#dagster.DagsterEventType.ASSET_WIPED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.CODE_LOCATION_UPDATED'>CODE_LOCATION_UPDATED `=` 'CODE_LOCATION_UPDATED'<a href="#dagster.DagsterEventType.CODE_LOCATION_UPDATED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.ENGINE_EVENT'>ENGINE_EVENT `=` 'ENGINE_EVENT'<a href="#dagster.DagsterEventType.ENGINE_EVENT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.FRESHNESS_STATE_CHANGE'>FRESHNESS_STATE_CHANGE `=` 'FRESHNESS_STATE_CHANGE'<a href="#dagster.DagsterEventType.FRESHNESS_STATE_CHANGE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.FRESHNESS_STATE_EVALUATION'>FRESHNESS_STATE_EVALUATION `=` 'FRESHNESS_STATE_EVALUATION'<a href="#dagster.DagsterEventType.FRESHNESS_STATE_EVALUATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.HANDLED_OUTPUT'>HANDLED_OUTPUT `=` 'HANDLED_OUTPUT'<a href="#dagster.DagsterEventType.HANDLED_OUTPUT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.HOOK_COMPLETED'>HOOK_COMPLETED `=` 'HOOK_COMPLETED'<a href="#dagster.DagsterEventType.HOOK_COMPLETED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.HOOK_ERRORED'>HOOK_ERRORED `=` 'HOOK_ERRORED'<a href="#dagster.DagsterEventType.HOOK_ERRORED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.HOOK_SKIPPED'>HOOK_SKIPPED `=` 'HOOK_SKIPPED'<a href="#dagster.DagsterEventType.HOOK_SKIPPED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.LOADED_INPUT'>LOADED_INPUT `=` 'LOADED_INPUT'<a href="#dagster.DagsterEventType.LOADED_INPUT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.LOGS_CAPTURED'>LOGS_CAPTURED `=` 'LOGS_CAPTURED'<a href="#dagster.DagsterEventType.LOGS_CAPTURED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.OBJECT_STORE_OPERATION'>OBJECT_STORE_OPERATION `=` 'OBJECT_STORE_OPERATION'<a href="#dagster.DagsterEventType.OBJECT_STORE_OPERATION" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_CANCELED'>PIPELINE_CANCELED `=` 'PIPELINE_CANCELED'<a href="#dagster.DagsterEventType.PIPELINE_CANCELED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_CANCELING'>PIPELINE_CANCELING `=` 'PIPELINE_CANCELING'<a href="#dagster.DagsterEventType.PIPELINE_CANCELING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_DEQUEUED'>PIPELINE_DEQUEUED `=` 'PIPELINE_DEQUEUED'<a href="#dagster.DagsterEventType.PIPELINE_DEQUEUED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_ENQUEUED'>PIPELINE_ENQUEUED `=` 'PIPELINE_ENQUEUED'<a href="#dagster.DagsterEventType.PIPELINE_ENQUEUED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_FAILURE'>PIPELINE_FAILURE `=` 'PIPELINE_FAILURE'<a href="#dagster.DagsterEventType.PIPELINE_FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_START'>PIPELINE_START `=` 'PIPELINE_START'<a href="#dagster.DagsterEventType.PIPELINE_START" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_STARTING'>PIPELINE_STARTING `=` 'PIPELINE_STARTING'<a href="#dagster.DagsterEventType.PIPELINE_STARTING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.PIPELINE_SUCCESS'>PIPELINE_SUCCESS `=` 'PIPELINE_SUCCESS'<a href="#dagster.DagsterEventType.PIPELINE_SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RESOURCE_INIT_FAILURE'>RESOURCE_INIT_FAILURE `=` 'RESOURCE_INIT_FAILURE'<a href="#dagster.DagsterEventType.RESOURCE_INIT_FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RESOURCE_INIT_STARTED'>RESOURCE_INIT_STARTED `=` 'RESOURCE_INIT_STARTED'<a href="#dagster.DagsterEventType.RESOURCE_INIT_STARTED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RESOURCE_INIT_SUCCESS'>RESOURCE_INIT_SUCCESS `=` 'RESOURCE_INIT_SUCCESS'<a href="#dagster.DagsterEventType.RESOURCE_INIT_SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_CANCELED'>RUN_CANCELED `=` 'PIPELINE_CANCELED'<a href="#dagster.DagsterEventType.RUN_CANCELED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_CANCELING'>RUN_CANCELING `=` 'PIPELINE_CANCELING'<a href="#dagster.DagsterEventType.RUN_CANCELING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_DEQUEUED'>RUN_DEQUEUED `=` 'PIPELINE_DEQUEUED'<a href="#dagster.DagsterEventType.RUN_DEQUEUED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_ENQUEUED'>RUN_ENQUEUED `=` 'PIPELINE_ENQUEUED'<a href="#dagster.DagsterEventType.RUN_ENQUEUED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_FAILURE'>RUN_FAILURE `=` 'PIPELINE_FAILURE'<a href="#dagster.DagsterEventType.RUN_FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_START'>RUN_START `=` 'PIPELINE_START'<a href="#dagster.DagsterEventType.RUN_START" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_STARTING'>RUN_STARTING `=` 'PIPELINE_STARTING'<a href="#dagster.DagsterEventType.RUN_STARTING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.RUN_SUCCESS'>RUN_SUCCESS `=` 'PIPELINE_SUCCESS'<a href="#dagster.DagsterEventType.RUN_SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_EXPECTATION_RESULT'>STEP_EXPECTATION_RESULT `=` 'STEP_EXPECTATION_RESULT'<a href="#dagster.DagsterEventType.STEP_EXPECTATION_RESULT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_FAILURE'>STEP_FAILURE `=` 'STEP_FAILURE'<a href="#dagster.DagsterEventType.STEP_FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_INPUT'>STEP_INPUT `=` 'STEP_INPUT'<a href="#dagster.DagsterEventType.STEP_INPUT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_OUTPUT'>STEP_OUTPUT `=` 'STEP_OUTPUT'<a href="#dagster.DagsterEventType.STEP_OUTPUT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_RESTARTED'>STEP_RESTARTED `=` 'STEP_RESTARTED'<a href="#dagster.DagsterEventType.STEP_RESTARTED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_SKIPPED'>STEP_SKIPPED `=` 'STEP_SKIPPED'<a href="#dagster.DagsterEventType.STEP_SKIPPED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_START'>STEP_START `=` 'STEP_START'<a href="#dagster.DagsterEventType.STEP_START" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_SUCCESS'>STEP_SUCCESS `=` 'STEP_SUCCESS'<a href="#dagster.DagsterEventType.STEP_SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_UP_FOR_RETRY'>STEP_UP_FOR_RETRY `=` 'STEP_UP_FOR_RETRY'<a href="#dagster.DagsterEventType.STEP_UP_FOR_RETRY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_WORKER_STARTED'>STEP_WORKER_STARTED `=` 'STEP_WORKER_STARTED'<a href="#dagster.DagsterEventType.STEP_WORKER_STARTED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterEventType.STEP_WORKER_STARTING'>STEP_WORKER_STARTING `=` 'STEP_WORKER_STARTING'<a href="#dagster.DagsterEventType.STEP_WORKER_STARTING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="reconstructable-jobs">
## Reconstructable jobs
<dl>
<dt><Link class="anchor" id='dagster.reconstructable'>`class` dagster.reconstructable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/reconstruct.py#L328' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.reconstructable" class="hash-link"></a></Link></dt>
<dd>
Create a `ReconstructableJob` from a
function that returns a [`JobDefinition`](jobs.mdx#dagster.JobDefinition)/[`JobDefinition`](jobs.mdx#dagster.JobDefinition),
or a function decorated with [`@job`](jobs.mdx#dagster.job).
When your job must cross process boundaries, e.g., for execution on multiple nodes or
in different systems (like `dagstermill`), Dagster must know how to reconstruct the job
on the other side of the process boundary.
Passing a job created with `~dagster.GraphDefinition.to_job` to `reconstructable()`,
requires you to wrap that job’s definition in a module-scoped function, and pass that function
instead:
```python
from dagster import graph, reconstructable
@graph
def my_graph():
...
def define_my_job():
return my_graph.to_job()
reconstructable(define_my_job)
```
This function implements a very conservative strategy for reconstruction, so that its behavior
is easy to predict, but as a consequence it is not able to reconstruct certain kinds of jobs
or jobs, such as those defined by lambdas, in nested scopes (e.g., dynamically within a method
call), or in interactive environments such as the Python REPL or Jupyter notebooks.
If you need to reconstruct objects constructed in these ways, you should use
`build_reconstructable_job()` instead, which allows you to
specify your own reconstruction strategy.
Examples:
```python
from dagster import job, reconstructable
@job
def foo_job():
...
reconstructable_foo_job = reconstructable(foo_job)
@graph
def foo():
...
def make_bar_job():
return foo.to_job()
reconstructable_bar_job = reconstructable(make_bar_job)
```
</dd>
</dl>
</div>
<div class="section" id="executors">
## Executors
<dl>
<dt><Link class="anchor" id='dagster.multi_or_in_process_executor'>dagster.multi_or_in_process_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L480' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multi_or_in_process_executor" class="hash-link"></a></Link></dt>
<dd>
The default executor for a job.
This is the executor available by default on a [`JobDefinition`](jobs.mdx#dagster.JobDefinition)
that does not provide custom executors. This executor has a multiprocessing-enabled mode, and a
single-process mode. By default, multiprocessing mode is enabled. Switching between multiprocess
mode and in-process mode can be achieved via config.
```yaml
execution:
config:
multiprocess:
execution:
config:
in_process:
```
When using the multiprocess mode, `max_concurrent` and `retries` can also be configured.
```yaml
execution:
config:
multiprocess:
max_concurrent: 4
retries:
enabled:
```
The `max_concurrent` arg is optional and tells the execution engine how many processes may run
concurrently. By default, or if you set `max_concurrent` to be 0, this is the return value of
`python:multiprocessing.cpu_count()`.
When using the in_process mode, then only retries can be configured.
Execution priority can be configured using the `dagster/priority` tag via op metadata,
where the higher the number the higher the priority. 0 is the default and both positive
and negative numbers can be used.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.in_process_executor'>dagster.in_process_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L292' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.in_process_executor" class="hash-link"></a></Link></dt>
<dd>
The in-process executor executes all steps in a single process.
To select it, include the following top-level fragment in config:
```yaml
execution:
in_process:
```
Execution priority can be configured using the `dagster/priority` tag via op metadata,
where the higher the number the higher the priority. 0 is the default and both positive
and negative numbers can be used.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.multiprocess_executor'>dagster.multiprocess_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L407' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multiprocess_executor" class="hash-link"></a></Link></dt>
<dd>
The multiprocess executor executes each step in an individual process.
Any job that does not specify custom executors will use the multiprocess_executor by default.
To configure the multiprocess executor, include a fragment such as the following in your run
config:
```yaml
execution:
config:
multiprocess:
max_concurrent: 4
```
The `max_concurrent` arg is optional and tells the execution engine how many processes may run
concurrently. By default, or if you set `max_concurrent` to be None or 0, this is the return value of
`python:multiprocessing.cpu_count()`.
Execution priority can be configured using the `dagster/priority` tag via op metadata,
where the higher the number the higher the priority. 0 is the default and both positive
and negative numbers can be used.
</dd>
</dl>
</div>
<div class="section" id="contexts">
## Contexts
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext'>`class` dagster.AssetExecutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.add_asset_metadata'>add_asset_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L478' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.add_asset_metadata" class="hash-link"></a></Link></dt>
<dd>
Add metadata to an asset materialization event. This metadata will be
available in the Dagster UI.
Parameters:
- <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The metadata to add to the asset materialization event.
- <strong>asset_key</strong> (<em>Optional</em><em>[</em><em>CoercibleToAssetKey</em><em>]</em>) – The asset key to add metadata to. Does not need to be provided if only one asset is currently being materialized.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The partition key to add metadata to, if applicable. Should not be provided on non-partitioned assets. If not provided on a partitioned asset, the metadata will be added to all partitions of the asset currently being materialized.
Examples:
Adding metadata to the asset materialization event for a single asset:
```python
import dagster as dg
@dg.asset
def my_asset(context):
# Add metadata
context.add_asset_metadata({"key": "value"})
```
Adding metadata to the asset materialization event for a particular partition of a partitioned asset:
```python
import dagster as dg
@dg.asset(partitions_def=dg.StaticPartitionsDefinition(["a", "b"]))
def my_asset(context):
# Adds metadata to all partitions currently being materialized, since no
# partition is specified.
context.add_asset_metadata({"key": "value"})
for partition_key in context.partition_keys:
# Add metadata only to the event for partition "a"
if partition_key == "a":
context.add_asset_metadata({"key": "value"}, partition_key=partition_key)
```
Adding metadata to the asset materialization event for a particular asset in a multi-asset.
```python
import dagster as dg
@dg.multi_asset(specs=[dg.AssetSpec("asset1"), dg.AssetSpec("asset2")])
def my_multi_asset(context):
# Add metadata to the materialization event for "asset1"
context.add_asset_metadata({"key": "value"}, asset_key="asset1")
# THIS line will fail since asset key is not specified:
context.add_asset_metadata({"key": "value"})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.add_output_metadata'>add_output_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L464' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.add_output_metadata" class="hash-link"></a></Link></dt>
<dd>
Add metadata to one of the outputs of an op.
This can be invoked multiple times per output in the body of an op. If the same key is
passed multiple times, the value associated with the last call will be used.
Parameters:
- <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The metadata to attach to the output
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output to attach metadata to. If there is only one output on the op, then this argument does not need to be provided. The metadata will automatically be attached to the only output.
- <strong>mapping_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The mapping key of the output to attach metadata to. If the output is not dynamic, this argument does not need to be provided.
<strong>Examples:</strong>
```python
from dagster import Out, op
from typing import Tuple
@op
def add_metadata(context):
context.add_output_metadata({"foo", "bar"})
return 5 # Since the default output is called "result", metadata will be attached to the output "result".
@op(out={"a": Out(), "b": Out()})
def add_metadata_two_outputs(context) -> Tuple[str, int]:
context.add_output_metadata({"foo": "bar"}, output_name="b")
context.add_output_metadata({"baz": "bat"}, output_name="a")
return ("dog", 5)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_key_for_input'>asset_key_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L343' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_key_for_input" class="hash-link"></a></Link></dt>
<dd>
Return the AssetKey for the corresponding input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_key_for_output'>asset_key_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L333' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_key_for_output" class="hash-link"></a></Link></dt>
<dd>
Return the AssetKey for the corresponding output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_key_for_input'>asset_partition_key_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L427' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_key_for_input" class="hash-link"></a></Link></dt>
<dd>
Returns the partition key of the upstream asset corresponding to the given input.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_for_input("self_dependent_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-20"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_key_for_output'>asset_partition_key_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L216' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_key_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method asset_partition_key_for_output on AssetExecutionContext. Use context.partition_key instead..
:::
Returns the asset partition key for the given output.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_for_output("first_asset"))
context.log.info(context.asset_partition_key_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
# "2023-08-21"
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_key_range_for_input'>asset_partition_key_range_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L422' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_key_range_for_input" class="hash-link"></a></Link></dt>
<dd>
Return the PartitionKeyRange for the corresponding input. Errors if the asset depends on a
non-contiguous chunk of the input.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_key_range_for_input` to get the range of partitions keys of the input that
are relevant to that backfill.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_range_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_range_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-20", end="2023-08-24")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_range_for_input("self_dependent_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-20", end="2023-08-24")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_key_range_for_output'>asset_partition_key_range_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L228' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_key_range_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method asset_partition_key_range_for_output on AssetExecutionContext. Use context.partition_key_range instead..
:::
Return the PartitionKeyRange for the corresponding output. Errors if the run is not partitioned.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_key_range_for_output` to get all of the partitions being materialized
by the backfill.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition key range for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_range_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_range_for_output("first_asset"))
context.log.info(context.asset_partition_key_range_for_output("second_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_range_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_keys_for_input'>asset_partition_keys_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L437' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_keys_for_input" class="hash-link"></a></Link></dt>
<dd>
Returns a list of the partition keys of the upstream asset corresponding to the
given input.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_keys_for_input` to get all of the partition keys of the input that
are relevant to that backfill.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_keys_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_keys_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-20", "2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24"]
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_keys_for_input("self_dependent_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-20", "2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_keys_for_output'>asset_partition_keys_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L242' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_keys_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method asset_partition_keys_for_output on AssetExecutionContext. Use context.partition_keys instead..
:::
Returns a list of the partition keys for the given output.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_keys_for_output` to get all of the partitions being materialized
by the backfill.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition keys for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_keys_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_keys_for_output("first_asset"))
context.log.info(context.asset_partition_keys_for_output("second_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_keys_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partitions_def_for_input'>asset_partitions_def_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L432' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partitions_def_for_input" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition on the upstream asset corresponding to this input.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the PartitionsDefinition for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def upstream_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_def_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partitions_def_for_output'>asset_partitions_def_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L236' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partitions_def_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method asset_partitions_def_for_output on AssetExecutionContext. Use context.assets_def.partitions_def instead..
:::
The PartitionsDefinition on the asset corresponding to this output.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the PartitionsDefinition for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_def_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_def_for_output("first_asset"))
context.log.info(context.asset_partitions_def_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
# DailyPartitionsDefinition("2023-08-20")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partitions_time_window_for_input'>asset_partitions_time_window_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L442' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partitions_time_window_for_input" class="hash-link"></a></Link></dt>
<dd>
The time window for the partitions of the input asset.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partitions_time_window_for_input` to get the time window of the input that
are relevant to that backfill.
Raises an error if either of the following are true:
- The input asset has no partitioning.
- The input asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_time_window_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_time_window_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-20", "2023-08-21")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partitions_time_window_for_input("self_dependent_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-20", "2023-08-21")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-20", "2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partitions_time_window_for_output'>asset_partitions_time_window_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partitions_time_window_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method asset_partitions_time_window_for_output on AssetExecutionContext. Use context.partition_time_window instead..
:::
The time window for the partitions of the output asset.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partitions_time_window_for_output` to get the TimeWindow of all of the partitions
being materialized by the backfill.
Raises an error if either of the following are true:
- The output asset has no partitioning.
- The output asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_time_window_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_time_window_for_output("first_asset"))
context.log.info(context.asset_partitions_time_window_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partitions_time_window_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.get_asset_provenance'>get_asset_provenance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L570' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.get_asset_provenance" class="hash-link"></a></Link></dt>
<dd>
Return the provenance information for the most recent materialization of an asset.
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – Key of the asset for which to retrieve provenance.Returns:
Provenance information for the most recent
materialization of the asset. Returns <cite>None</cite> if the asset was never materialized or
the materialization record is too old to contain provenance information.
Return type: Optional[DataProvenance]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.get_mapping_key'>get_mapping_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L273' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.get_mapping_key" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method get_mapping_key on AssetExecutionContext. Use context.op_execution_context.get_mapping_key instead..
:::
Which mapping_key this execution is for if downstream of a DynamicOutput, otherwise None.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.get_tag'>get_tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L206' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.get_tag" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method get_tag on AssetExecutionContext. Use context.run.tags.get(key) instead..
:::
Get a logging tag.
Parameters: <strong>key</strong> (<em>tag</em>) – The tag to get.Returns: The value of the tag, if present.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.has_tag'>has_tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L200' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.has_tag" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method has_tag on AssetExecutionContext. Use key in context.run.tags instead..
:::
Check if a logging tag is set.
Parameters: <strong>key</strong> (<em>str</em>) – The tag to check.Returns: Whether the tag is set.Return type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.log_event'>log_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L457' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.log_event" class="hash-link"></a></Link></dt>
<dd>
Log an AssetMaterialization, AssetObservation, or ExpectationResult from within the body of an op.
Events logged with this method will appear in the list of DagsterEvents, as well as the event log.
Parameters: <strong>event</strong> (<em>Union</em><em>[</em>[*AssetMaterialization*](ops.mdx#dagster.AssetMaterialization)<em>, </em>[*AssetObservation*](assets.mdx#dagster.AssetObservation)<em>, </em>[*ExpectationResult*](ops.mdx#dagster.ExpectationResult)<em>]</em>) – The event to log.
<strong>Examples:</strong>
```python
from dagster import op, AssetMaterialization
@op
def log_materialization(context):
context.log_event(AssetMaterialization("foo"))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.output_for_asset_key'>output_for_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L338' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.output_for_asset_key" class="hash-link"></a></Link></dt>
<dd>
Return the output name for the corresponding asset key.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L315' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_key" class="hash-link"></a></Link></dt>
<dd>
The AssetKey for the current asset. In a multi_asset, use asset_key_for_output instead.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.asset_partition_key_range'>`property` asset_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L403' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.asset_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_key_range` instead..
:::
The range of partition keys for the current run.
If run is for a single partition key, return a <cite>PartitionKeyRange</cite> with the same start and
end. Raises an error if the current run is not a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.assets_def'>`property` assets_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L327' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.assets_def" class="hash-link"></a></Link></dt>
<dd>
The backing AssetsDefinition for what is currently executing, errors if not available.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.has_assets_def'>`property` has_assets_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L321' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.has_assets_def" class="hash-link"></a></Link></dt>
<dd>
If there is a backing AssetsDefinition for what is currently executing.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.has_partition_key'>`property` has_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L373' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.has_partition_key" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a single partition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.has_partition_key_range'>`property` has_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L397' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.has_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a range of partitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L356' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.instance" class="hash-link"></a></Link></dt>
<dd>
The current Dagster instance.
Type: [DagsterInstance](internals.mdx#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L160' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.job_def" class="hash-link"></a></Link></dt>
<dd>
The definition for the currently executing job. Information like the job name, and job tags
can be found on the JobDefinition.
Returns: JobDefinition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L307' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the currently executing pipeline.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L120' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.log" class="hash-link"></a></Link></dt>
<dd>
The log manager available in the execution context. Logs will be viewable in the Dagster UI.
Returns: DagsterLogManager.
Example:
```python
@asset
def logger(context):
context.log.info("Info level message")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.multi_partition_key'>`property` multi_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L385' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.multi_partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run, typed as a [`MultiPartitionKey`](partitions.mdx#dagster.MultiPartitionKey).
Raises an error if the current run is not a partitioned run, or if the partitions definition
is not a [`MultiPartitionsDefinition`](partitions.mdx#dagster.MultiPartitionsDefinition).
The returned `MultiPartitionKey` provides a
`keys_by_dimension` property for accessing individual
dimension values.
Examples:
```python
partitions_def = MultiPartitionsDefinition(
{
"date": DailyPartitionsDefinition("2023-08-20"),
"color": StaticPartitionsDefinition(["red", "blue"]),
}
)
@asset(partitions_def=partitions_def)
def my_asset(context: AssetExecutionContext):
key = context.multi_partition_key
context.log.info(key.keys_by_dimension)
# materializing the 2023-08-21|red partition of this asset will log:
# {"date": "2023-08-21", "color": "red"}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.op_config'>`property` op_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L248' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.op_config" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method op_config on AssetExecutionContext. Use context.op_execution_context.op_config instead..
:::
The parsed config specific to this op.
Type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L299' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The current op definition.
Type: [OpDefinition](ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L379' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run.
Raises an error if the current run is not a partitioned run. Or if the current run is operating
over a range of partitions (ie. a backfill of several partitions executed in a single run).
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key)
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.partition_key_range'>`property` partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L410' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The range of partition keys for the current run.
If run is for a single partition key, returns a <cite>PartitionKeyRange</cite> with the same start and
end. Raises an error if the current run is not a partitioned run.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key_range)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.partition_keys'>`property` partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L391' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of the partition keys for the current run.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `partition_keys` to get all of the partitions being materialized
by the backfill.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(partitions_def=partitions_def)
def an_asset(context: AssetExecutionContext):
context.log.info(context.partition_keys)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.partition_time_window'>`property` partition_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L416' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.partition_time_window" class="hash-link"></a></Link></dt>
<dd>
The partition time window for the current run.
Raises an error if the current run is not a partitioned run, or if the job’s partition
definition is not a TimeWindowPartitionsDefinition.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_time_window)
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.pdb'>`property` pdb <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.pdb" class="hash-link"></a></Link></dt>
<dd>
Gives access to pdb debugging from within the asset. Materializing the asset via the
Dagster UI or CLI will enter the pdb debugging context in the process used to launch the UI or
run the CLI.
Returns: dagster.utils.forked_pdb.ForkedPdb
Example:
```python
@asset
def debug(context):
context.pdb.set_trace()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L583' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.resources" class="hash-link"></a></Link></dt>
<dd>
The currently available resources.
Type: Resources
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.selected_asset_check_keys'>`property` selected_asset_check_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L562' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.selected_asset_check_keys" class="hash-link"></a></Link></dt>
<dd>
Get the asset check keys that correspond to the current selection of assets this execution is expected to materialize.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.selected_asset_keys'>`property` selected_asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_execution_context.py#L348' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.selected_asset_keys" class="hash-link"></a></Link></dt>
<dd>
Get the set of AssetKeys this execution is expected to materialize.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetExecutionContext.selected_output_names'>`property` selected_output_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L279' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetExecutionContext.selected_output_names" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version a future release.
You have called the deprecated method selected_output_names on AssetExecutionContext. Use context.op_execution_context.selected_output_names instead..
:::
Get the output names that correspond to the current selection of assets this execution is expected to materialize.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext'>`class` dagster.OpExecutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object that can be made available as the first argument to the function
used for computing an op or asset.
This context object provides system information such as resources, config, and logging.
To construct an execution context for testing purposes, use [`dagster.build_op_context()`](#dagster.build_op_context).
Example:
```python
from dagster import op, OpExecutionContext
@op
def hello_world(context: OpExecutionContext):
context.log.info("Hello, world!")
```
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.add_output_metadata'>add_output_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L505' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.add_output_metadata" class="hash-link"></a></Link></dt>
<dd>
Add metadata to one of the outputs of an op.
This can be invoked multiple times per output in the body of an op. If the same key is
passed multiple times, the value associated with the last call will be used.
Parameters:
- <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The metadata to attach to the output
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output to attach metadata to. If there is only one output on the op, then this argument does not need to be provided. The metadata will automatically be attached to the only output.
- <strong>mapping_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The mapping key of the output to attach metadata to. If the output is not dynamic, this argument does not need to be provided.
<strong>Examples:</strong>
```python
from dagster import Out, op
from typing import Tuple
@op
def add_metadata(context):
context.add_output_metadata({"foo", "bar"})
return 5 # Since the default output is called "result", metadata will be attached to the output "result".
@op(out={"a": Out(), "b": Out()})
def add_metadata_two_outputs(context) -> Tuple[str, int]:
context.add_output_metadata({"foo": "bar"}, output_name="b")
context.add_output_metadata({"baz": "bat"}, output_name="a")
return ("dog", 5)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_key_for_input'>asset_key_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L665' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_key_for_input" class="hash-link"></a></Link></dt>
<dd>
Return the AssetKey for the corresponding input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_key_for_output'>asset_key_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L645' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_key_for_output" class="hash-link"></a></Link></dt>
<dd>
Return the AssetKey for the corresponding output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_key_for_input'>asset_partition_key_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L937' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_key_for_input" class="hash-link"></a></Link></dt>
<dd>
Returns the partition key of the upstream asset corresponding to the given input.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_for_input("self_dependent_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-20"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_key_for_output'>asset_partition_key_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L676' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_key_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_key` instead..
:::
Returns the asset partition key for the given output.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_for_output("first_asset"))
context.log.info(context.asset_partition_key_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
# "2023-08-21"
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_key_range_for_input'>asset_partition_key_range_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L874' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_key_range_for_input" class="hash-link"></a></Link></dt>
<dd>
Return the PartitionKeyRange for the corresponding input. Errors if the asset depends on a
non-contiguous chunk of the input.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_key_range_for_input` to get the range of partitions keys of the input that
are relevant to that backfill.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_range_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_key_range_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-20", end="2023-08-24")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_range_for_input("self_dependent_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-20", end="2023-08-24")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_key_range_for_output'>asset_partition_key_range_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L810' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_key_range_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_key_range` instead..
:::
Return the PartitionKeyRange for the corresponding output. Errors if the run is not partitioned.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_key_range_for_output` to get all of the partitions being materialized
by the backfill.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition key range for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_range_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_key_range_for_output("first_asset"))
context.log.info(context.asset_partition_key_range_for_output("second_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_key_range_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_keys_for_input'>asset_partition_keys_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L1132' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_keys_for_input" class="hash-link"></a></Link></dt>
<dd>
Returns a list of the partition keys of the upstream asset corresponding to the
given input.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_keys_for_input` to get all of the partition keys of the input that
are relevant to that backfill.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_keys_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partition_keys_for_input("upstream_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-20", "2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24"]
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_keys_for_input("self_dependent_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-20", "2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_keys_for_output'>asset_partition_keys_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L1068' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_keys_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_keys` instead..
:::
Returns a list of the partition keys for the given output.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partition_keys_for_output` to get all of the partitions being materialized
by the backfill.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the partition keys for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_keys_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partition_keys_for_output("first_asset"))
context.log.info(context.asset_partition_keys_for_output("second_asset"))
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partition_keys_for_output())
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partitions_def_for_input'>asset_partitions_def_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L1030' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partitions_def_for_input" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition on the upstream asset corresponding to this input.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the PartitionsDefinition for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def upstream_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_def_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partitions_def_for_output'>asset_partitions_def_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L980' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partitions_def_for_output" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition on the asset corresponding to this output.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the PartitionsDefinition for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_def_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_def_for_output("first_asset"))
context.log.info(context.asset_partitions_def_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# DailyPartitionsDefinition("2023-08-20")
# DailyPartitionsDefinition("2023-08-20")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partitions_time_window_for_input'>asset_partitions_time_window_for_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L1197' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partitions_time_window_for_input" class="hash-link"></a></Link></dt>
<dd>
The time window for the partitions of the input asset.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partitions_time_window_for_input` to get the time window of the input that
are relevant to that backfill.
Raises an error if either of the following are true:
- The input asset has no partitioning.
- The input asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
Parameters: <strong>input_name</strong> (<em>str</em>) – The name of the input to get the partition key for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def upstream_asset():
...
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_time_window_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
ins={
"upstream_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
},
partitions_def=partitions_def,
)
def another_asset(context: AssetExecutionContext, upstream_asset):
context.log.info(context.asset_partitions_time_window_for_input("upstream_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-20", "2023-08-21")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partitions_time_window_for_input("self_dependent_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-20", "2023-08-21")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-20", "2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partitions_time_window_for_output'>asset_partitions_time_window_for_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L734' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partitions_time_window_for_output" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_time_window` instead..
:::
The time window for the partitions of the output asset.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `asset_partitions_time_window_for_output` to get the TimeWindow of all of the partitions
being materialized by the backfill.
Raises an error if either of the following are true:
- The output asset has no partitioning.
- The output asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
Parameters: <strong>output_name</strong> (<em>str</em>) – For assets defined with the `@asset` decorator, the name of the output
will be automatically provided. For assets defined with `@multi_asset`, `output_name`
should be the op output associated with the asset key (as determined by AssetOut)
to get the time window for.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def an_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_time_window_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
@multi_asset(
outs={
"first_asset": AssetOut(key=["my_assets", "first_asset"]),
"second_asset": AssetOut(key=["my_assets", "second_asset"]),
},
partitions_def=partitions_def,
)
def a_multi_asset(context: AssetExecutionContext):
context.log.info(context.asset_partitions_time_window_for_output("first_asset"))
context.log.info(context.asset_partitions_time_window_for_output("second_asset"))
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
# TimeWindow("2023-08-21", "2023-08-26")
@asset(
partitions_def=partitions_def,
ins={
"self_dependent_asset": AssetIn(partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)),
}
)
def self_dependent_asset(context: AssetExecutionContext, self_dependent_asset):
context.log.info(context.asset_partitions_time_window_for_output())
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# TimeWindow("2023-08-21", "2023-08-26")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.get_asset_provenance'>get_asset_provenance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L1273' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.get_asset_provenance" class="hash-link"></a></Link></dt>
<dd>
Return the provenance information for the most recent materialization of an asset.
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – Key of the asset for which to retrieve provenance.Returns:
Provenance information for the most recent
materialization of the asset. Returns <cite>None</cite> if the asset was never materialized or
the materialization record is too old to contain provenance information.
Return type: Optional[DataProvenance]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.get_mapping_key'>get_mapping_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L578' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.get_mapping_key" class="hash-link"></a></Link></dt>
<dd>
Which mapping_key this execution is for if downstream of a DynamicOutput, otherwise None.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.get_tag'>get_tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.get_tag" class="hash-link"></a></Link></dt>
<dd>
Get a logging tag.
Parameters: <strong>key</strong> (<em>tag</em>) – The tag to get.Returns: The value of the tag, if present.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.has_tag'>has_tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L428' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.has_tag" class="hash-link"></a></Link></dt>
<dd>
Check if a logging tag is set.
Parameters: <strong>key</strong> (<em>str</em>) – The tag to check.Returns: Whether the tag is set.Return type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.log_event'>log_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L469' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.log_event" class="hash-link"></a></Link></dt>
<dd>
Log an AssetMaterialization, AssetObservation, or ExpectationResult from within the body of an op.
Events logged with this method will appear in the list of DagsterEvents, as well as the event log.
Parameters: <strong>event</strong> (<em>Union</em><em>[</em>[*AssetMaterialization*](ops.mdx#dagster.AssetMaterialization)<em>, </em>[*AssetObservation*](assets.mdx#dagster.AssetObservation)<em>, </em>[*ExpectationResult*](ops.mdx#dagster.ExpectationResult)<em>]</em>) – The event to log.
<strong>Examples:</strong>
```python
from dagster import op, AssetMaterialization
@op
def log_materialization(context):
context.log_event(AssetMaterialization("foo"))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.output_for_asset_key'>output_for_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L656' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.output_for_asset_key" class="hash-link"></a></Link></dt>
<dd>
Return the output name for the corresponding asset key.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L587' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_key" class="hash-link"></a></Link></dt>
<dd>
The AssetKey for the current asset. In a multi_asset, use asset_key_for_output instead.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.asset_partition_key_range'>`property` asset_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L363' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.asset_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `partition_key_range` instead..
:::
The range of partition keys for the current run.
If run is for a single partition key, return a <cite>PartitionKeyRange</cite> with the same start and
end. Raises an error if the current run is not a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.assets_def'>`property` assets_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L605' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.assets_def" class="hash-link"></a></Link></dt>
<dd>
The backing AssetsDefinition for what is currently executing, errors if not available.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.has_assets_def'>`property` has_assets_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L598' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.has_assets_def" class="hash-link"></a></Link></dt>
<dd>
If there is a backing AssetsDefinition for what is currently executing.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.has_partition_key'>`property` has_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L269' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.has_partition_key" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a single partition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.has_partition_key_range'>`property` has_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L374' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.has_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a range of partitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.has_partitions'>`property` has_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L263' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.has_partitions" class="hash-link"></a></Link></dt>
<dd>
Whether the current run is a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L149' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.instance" class="hash-link"></a></Link></dt>
<dd>
The current Dagster instance.
Type: [DagsterInstance](internals.mdx#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L209' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.job_def" class="hash-link"></a></Link></dt>
<dd>
The currently executing job.
Type: [JobDefinition](jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the currently executing pipeline.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.location_name'>`property` location_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L189' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.location_name" class="hash-link"></a></Link></dt>
<dd>
The name of the code location for this run.
This is the name of the code location (repository location) from which
the job/pipeline was loaded. Returns None if the job was not loaded from
a remote code location (e.g., in tests or when running locally without
a code location).
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L226' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.log" class="hash-link"></a></Link></dt>
<dd>
The log manager available in the execution context.
Type: [DagsterLogManager](loggers.mdx#dagster.DagsterLogManager)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.multi_partition_key'>`property` multi_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L299' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.multi_partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run, typed as a [`MultiPartitionKey`](partitions.mdx#dagster.MultiPartitionKey).
Raises an error if the current run is not a partitioned run, or if the partitions definition
is not a [`MultiPartitionsDefinition`](partitions.mdx#dagster.MultiPartitionsDefinition).
The returned `MultiPartitionKey` provides a
`keys_by_dimension` property for accessing individual
dimension values.
Examples:
```python
partitions_def = MultiPartitionsDefinition(
{
"date": DailyPartitionsDefinition("2023-08-20"),
"color": StaticPartitionsDefinition(["red", "blue"]),
}
)
@asset(partitions_def=partitions_def)
def my_asset(context: AssetExecutionContext):
key = context.multi_partition_key
context.log.info(key.keys_by_dimension)
# materializing the 2023-08-21|red partition of this asset will log:
# {"date": "2023-08-21", "color": "red"}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.op_config'>`property` op_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L132' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.op_config" class="hash-link"></a></Link></dt>
<dd>
The parsed config specific to this op.
Type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L257' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The current op definition.
Type: [OpDefinition](ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L275' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run.
Raises an error if the current run is not a partitioned run. Or if the current run is operating
over a range of partitions (ie. a backfill of several partitions executed in a single run).
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key)
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.partition_key_range'>`property` partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L380' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The range of partition keys for the current run.
If run is for a single partition key, returns a <cite>PartitionKeyRange</cite> with the same start and
end. Raises an error if the current run is not a partitioned run.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key_range)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.partition_keys'>`property` partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L331' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of the partition keys for the current run.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `partition_keys` to get all of the partitions being materialized
by the backfill.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(partitions_def=partitions_def)
def an_asset(context: AssetExecutionContext):
context.log.info(context.partition_keys)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.partition_time_window'>`property` partition_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L404' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.partition_time_window" class="hash-link"></a></Link></dt>
<dd>
The partition time window for the current run.
Raises an error if the current run is not a partitioned run, or if the job’s partition
definition is not a TimeWindowPartitionsDefinition.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_time_window)
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.pdb'>`property` pdb <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L155' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.pdb" class="hash-link"></a></Link></dt>
<dd>
Gives access to pdb debugging from within the op.
Example:
```python
@op
def debug(context):
context.pdb.set_trace()
```
Type: dagster.utils.forked_pdb.ForkedPdb
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L172' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.resources" class="hash-link"></a></Link></dt>
<dd>
The currently available resources.
Type: Resources
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.retry_number'>`property` retry_number <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L569' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.retry_number" class="hash-link"></a></Link></dt>
<dd>
Which retry attempt is currently executing i.e. 0 for initial attempt, 1 for first retry, etc.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.run'>`property` run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L143' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.run" class="hash-link"></a></Link></dt>
<dd>
The current run.
Type: [DagsterRun](internals.mdx#dagster.DagsterRun)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.run_config'>`property` run_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L203' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.run_config" class="hash-link"></a></Link></dt>
<dd>
The run config for the current execution.
Type: dict
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L183' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The id of the current execution’s run.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.selected_asset_check_keys'>`property` selected_asset_check_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L633' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.selected_asset_check_keys" class="hash-link"></a></Link></dt>
<dd>
Get the asset check keys that correspond to the current selection of assets this execution is expected to materialize.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.selected_asset_keys'>`property` selected_asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L616' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.selected_asset_keys" class="hash-link"></a></Link></dt>
<dd>
Get the set of AssetKeys this execution is expected to materialize.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpExecutionContext.selected_output_names'>`property` selected_output_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/op_execution_context.py#L639' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpExecutionContext.selected_output_names" class="hash-link"></a></Link></dt>
<dd>
Get the output names that correspond to the current selection of assets this execution is expected to materialize.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_op_context'>dagster.build_op_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/invocation.py#L941' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_op_context" class="hash-link"></a></Link></dt>
<dd>
Builds op execution context from provided parameters.
`build_op_context` can be used as either a function or context manager. If there is a
provided resource that is a context manager, then `build_op_context` must be used as a
context manager. This function can be used to provide the context argument when directly
invoking a op.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to provide to the context. These can be either values or resource definitions.
- <strong>op_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the op.
- <strong>resources_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the resources.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured for the context. Defaults to DagsterInstance.ephemeral().
- <strong>mapping_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A key representing the mapping key from an upstream dynamic output. Can be accessed using `context.get_mapping_key()`.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – String value representing partition key to execute with.
- <strong>partition_key_range</strong> (<em>Optional</em><em>[</em>[*PartitionKeyRange*](partitions.mdx#dagster.PartitionKeyRange)<em>]</em>) – Partition key range to execute with.
- <strong>run_tags</strong> – Optional[Mapping[str, str]]: The tags for the executing run.
- <strong>event_loop</strong> – Optional[AbstractEventLoop]: An event loop for handling resources with async context managers.
Examples:
```python
context = build_op_context()
op_to_invoke(context)
with build_op_context(resources={"foo": context_manager_resource}) as context:
op_to_invoke(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_asset_context'>dagster.build_asset_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/invocation.py#L1042' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_asset_context" class="hash-link"></a></Link></dt>
<dd>
Builds asset execution context from provided parameters.
`build_asset_context` can be used as either a function or context manager. If there is a
provided resource that is a context manager, then `build_asset_context` must be used as a
context manager. This function can be used to provide the context argument when directly
invoking an asset.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to provide to the context. These can be either values or resource definitions.
- <strong>resources_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the resources.
- <strong>asset_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the asset.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured for the context. Defaults to DagsterInstance.ephemeral().
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – String value representing partition key to execute with.
- <strong>partition_key_range</strong> (<em>Optional</em><em>[</em>[*PartitionKeyRange*](partitions.mdx#dagster.PartitionKeyRange)<em>]</em>) – Partition key range to execute with.
- <strong>run_tags</strong> – Optional[Mapping[str, str]]: The tags for the executing run.
- <strong>event_loop</strong> – Optional[AbstractEventLoop]: An event loop for handling resources with async context managers.
Examples:
```python
context = build_asset_context()
asset_to_invoke(context)
with build_asset_context(resources={"foo": context_manager_resource}) as context:
asset_to_invoke(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext'>`class` dagster.AssetCheckExecutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.check_specs'>`property` check_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L103' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.check_specs" class="hash-link"></a></Link></dt>
<dd>
The asset check specs for the currently executing asset check.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.has_partition_key'>`property` has_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L144' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.has_partition_key" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a single partition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.has_partition_key_range'>`property` has_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L168' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.has_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
Whether the current run targets a range of partitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L128' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.instance" class="hash-link"></a></Link></dt>
<dd>
The current Dagster instance.
Type: [DagsterInstance](internals.mdx#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L82' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.job_def" class="hash-link"></a></Link></dt>
<dd>
The definition for the currently executing job. Information like the job name, and job tags
can be found on the JobDefinition.
Returns: JobDefinition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L56' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.log" class="hash-link"></a></Link></dt>
<dd>
The log manager available in the execution context. Logs will be viewable in the Dagster UI.
Returns: DagsterLogManager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.multi_partition_key'>`property` multi_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L156' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.multi_partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run, typed as a [`MultiPartitionKey`](partitions.mdx#dagster.MultiPartitionKey).
Raises an error if the current run is not a partitioned run, or if the partitions definition
is not a [`MultiPartitionsDefinition`](partitions.mdx#dagster.MultiPartitionsDefinition).
The returned `MultiPartitionKey` provides a
`keys_by_dimension` property for accessing individual
dimension values.
Examples:
```python
partitions_def = MultiPartitionsDefinition(
{
"date": DailyPartitionsDefinition("2023-08-20"),
"color": StaticPartitionsDefinition(["red", "blue"]),
}
)
@asset(partitions_def=partitions_def)
def my_asset(context: AssetExecutionContext):
key = context.multi_partition_key
context.log.info(key.keys_by_dimension)
# materializing the 2023-08-21|red partition of this asset will log:
# {"date": "2023-08-21", "color": "red"}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L120' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The current op definition.
Type: [OpDefinition](ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L150' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run.
Raises an error if the current run is not a partitioned run. Or if the current run is operating
over a range of partitions (ie. a backfill of several partitions executed in a single run).
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key)
# materializing the 2023-08-21 partition of this asset will log:
# "2023-08-21"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.partition_key_range'>`property` partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L174' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The range of partition keys for the current run.
If run is for a single partition key, returns a <cite>PartitionKeyRange</cite> with the same start and
end. Raises an error if the current run is not a partitioned run.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_key_range)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# PartitionKeyRange(start="2023-08-21", end="2023-08-25")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.partition_keys'>`property` partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L162' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of the partition keys for the current run.
If you want to write your asset to support running a backfill of several partitions in a single run,
you can use `partition_keys` to get all of the partitions being materialized
by the backfill.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(partitions_def=partitions_def)
def an_asset(context: AssetExecutionContext):
context.log.info(context.partition_keys)
# running a backfill of the 2023-08-21 through 2023-08-25 partitions of this asset will log:
# ["2023-08-21", "2023-08-22", "2023-08-23", "2023-08-24", "2023-08-25"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.partition_time_window'>`property` partition_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L180' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.partition_time_window" class="hash-link"></a></Link></dt>
<dd>
The partition time window for the current run.
Raises an error if the current run is not a partitioned run, or if the job’s partition
definition is not a TimeWindowPartitionsDefinition.
Examples:
```python
partitions_def = DailyPartitionsDefinition("2023-08-20")
@asset(
partitions_def=partitions_def
)
def my_asset(context: AssetExecutionContext):
context.log.info(context.partition_time_window)
# materializing the 2023-08-21 partition of this asset will log:
# TimeWindow("2023-08-21", "2023-08-22")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.pdb'>`property` pdb <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L64' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.pdb" class="hash-link"></a></Link></dt>
<dd>
Gives access to pdb debugging from within the asset. Materializing the asset via the
Dagster UI or CLI will enter the pdb debugging context in the process used to launch the UI or
run the CLI.
Returns: dagster.utils.forked_pdb.ForkedPdb
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L188' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.resources" class="hash-link"></a></Link></dt>
<dd>
The currently available resources.
Type: Resources
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetCheckExecutionContext.selected_asset_check_keys'>`property` selected_asset_check_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/asset_check_execution_context.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetCheckExecutionContext.selected_asset_check_keys" class="hash-link"></a></Link></dt>
<dd>
Get the asset check keys that correspond to the current selection of assets this execution is expected to materialize.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_asset_check_context'>dagster.build_asset_check_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/invocation.py#L1009' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_asset_check_context" class="hash-link"></a></Link></dt>
<dd>
Builds an asset check execution context from provided parameters.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to provide to the context. These can be either values or resource definitions.
- <strong>resources_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the resources.
- <strong>asset_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The config to provide to the asset.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured for the context. Defaults to DagsterInstance.ephemeral().
Examples:
```python
context = build_asset_check_context()
asset_check_to_invoke(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TypeCheckContext'>`class` dagster.TypeCheckContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1327' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TypeCheckContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object available to a type check function on a DagsterType.
<dl>
<dt><Link class="anchor" id='dagster.TypeCheckContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1354' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TypeCheckContext.log" class="hash-link"></a></Link></dt>
<dd>
Centralized log dispatch from user code.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TypeCheckContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1342' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TypeCheckContext.resources" class="hash-link"></a></Link></dt>
<dd>
An object whose attributes contain the resources available to this op.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TypeCheckContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1348' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TypeCheckContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The id of this job run.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="job-configuration">
## Job configuration
<dl>
<dt><Link class="anchor" id='dagster.validate_run_config'>dagster.validate_run_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/validate_run_config.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.validate_run_config" class="hash-link"></a></Link></dt>
<dd>
Function to validate a provided run config blob against a given job.
If validation is successful, this function will return a dictionary representation of the
validated config actually used during execution.
Parameters:
- <strong>job_def</strong> ([*JobDefinition*](jobs.mdx#dagster.JobDefinition)) – The job definition to validate run config against
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The run config to validate
Returns: A dictionary representation of the validated config.Return type: Dict[str, Any]
</dd>
</dl>
<div class="section" id="run-config-schema">
<Link id="config-schema"></Link>
### Run config schema
>
The `run_config` used for jobs has the following schema:
```default
{
# configuration for execution, required if executors require config
execution: {
# the name of one, and only one available executor, typically 'in_process' or 'multiprocess'
__executor_name__: {
# executor-specific config, if required or permitted
config: {
...
}
}
},
# configuration for loggers, required if loggers require config
loggers: {
# the name of an available logger
__logger_name__: {
# logger-specific config, if required or permitted
config: {
...
}
},
...
},
# configuration for resources, required if resources require config
resources: {
# the name of a resource
__resource_name__: {
# resource-specific config, if required or permitted
config: {
...
}
},
...
},
# configuration for underlying ops, required if ops require config
ops: {
# these keys align with the names of the ops, or their alias in this job
__op_name__: {
# pass any data that was defined via config_field
config: ...,
# configurably specify input values, keyed by input name
inputs: {
__input_name__: {
# if an dagster_type_loader is specified, that schema must be satisfied here;
# scalar, built-in types will generally allow their values to be specified directly:
value: ...
}
},
}
},
}
```
</div></div></div>
---
---
title: 'external assets instance api'
sidebar_position: 1000
title_meta: 'external assets instance api API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'external assets instance api Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="external-assets-instance-api">
# External assets instance API
As Dagster doesn’t control scheduling or materializing [external assets](https://docs.dagster.io/guides/build/assets/external-assets), it’s up to you to keep their metadata updated. The APIs in this reference can be used to keep external assets updated in Dagster.
<div class="section" id="instance-api">
## Instance API
External asset events can be recorded using `DagsterInstance.report_runless_asset_event()` on `DagsterInstance`.
<strong>Example:</strong> Reporting an asset materialization:
```python
from dagster import DagsterInstance, AssetMaterialization, AssetKey
instance = DagsterInstance.get()
instance.report_runless_asset_event(AssetMaterialization(AssetKey("example_asset")))
```
<strong>Example:</strong> Reporting an asset check evaluation:
```python
from dagster import DagsterInstance, AssetCheckEvaluation, AssetCheckKey
instance = DagsterInstance.get()
instance.report_runless_asset_event(
AssetCheckEvaluation(
asset_key=AssetKey("example_asset"),
check_name="example_check",
passed=True
)
)
```
</div>
<div class="section" id="rest-api">
## REST API
Refer to the [External assets REST API reference](https://docs.dagster.io/api/rest-api) for information and examples on the available APIs.
</div></div>
---
---
title: 'graphs'
sidebar_position: 1000
title_meta: 'graphs API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'graphs Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="graphs">
# Graphs
The core of a job is a _graph_ of ops - connected via data dependencies.
<dl>
<dt><Link class="anchor" id='dagster.graph'>@dagster.graph <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/graph_decorator.py#L125' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.graph" class="hash-link"></a></Link></dt>
<dd>
Create an op graph with the specified parameters from the decorated composition function.
Using this decorator allows you to build up a dependency graph by writing a
function that invokes ops (or other graphs) and passes the output to subsequent invocations.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op graph. Must be unique within any [`RepositoryDefinition`](repositories.mdx#dagster.RepositoryDefinition) containing the graph.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the graph.
- <strong>input_defs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>InputDefinition</em><em>]</em><em>]</em>) –
Information about the inputs that this graph maps. Information provided here will be combined with what can be inferred from the function signature, with these explicit InputDefinitions taking precedence.
- <strong>output_defs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>OutputDefinition</em><em>]</em><em>]</em>) –
Output definitions for the graph. If not provided explicitly, these will be inferred from typehints.
Uses of these outputs in the body of the decorated composition function, as well as the return value of the decorated function, will be used to infer the appropriate set of [`OutputMappings`](#dagster.OutputMapping) for the underlying [`GraphDefinition`](#dagster.GraphDefinition).
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*GraphIn*](#dagster.GraphIn)<em>]</em><em>]</em>) – Information about the inputs that this graph maps. Information provided here will be combined with what can be inferred from the function signature, with these explicit GraphIn taking precedence.
- <strong>out</strong> –
Information about the outputs that this graph maps. Information provided here will be combined with what can be inferred from the return type signature if the function does not use yield.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition'>`class` dagster.GraphDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/node_definition.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a Dagster op graph.
An op graph is made up of
- Nodes, which can either be an op (the functional unit of computation), or another graph.
- Dependencies, which determine how the values produced by nodes as outputs flow from one node to another. This tells Dagster how to arrange nodes into a directed, acyclic graph (DAG) of compute.
End users should prefer the [`@graph`](#dagster.graph) decorator. GraphDefinition is generally
intended to be used by framework authors or for programatically generated graphs.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the graph. Must be unique within any [`GraphDefinition`](#dagster.GraphDefinition) or [`JobDefinition`](jobs.mdx#dagster.JobDefinition) containing the graph.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the job.
- <strong>node_defs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>NodeDefinition</em><em>]</em><em>]</em>) – The set of ops / graphs used in this graph.
- <strong>dependencies</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em>[*NodeInvocation*](#dagster.NodeInvocation)<em>]</em><em>, </em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*DependencyDefinition*](#dagster.DependencyDefinition)<em>]</em><em>]</em><em>]</em>) – A structure that declares the dependencies of each op’s inputs on the outputs of other ops in the graph. Keys of the top level dict are either the string names of ops in the graph or, in the case of aliased ops, [`NodeInvocations`](#dagster.NodeInvocation). Values of the top level dict are themselves dicts, which map input names belonging to the op or aliased op to [`DependencyDefinitions`](#dagster.DependencyDefinition).
- <strong>input_mappings</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*InputMapping*](#dagster.InputMapping)<em>]</em><em>]</em>) – Defines the inputs to the nested graph, and how they map to the inputs of its constituent ops.
- <strong>output_mappings</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*OutputMapping*](#dagster.OutputMapping)<em>]</em><em>]</em>) – Defines the outputs of the nested graph, and how they map from the outputs of its constituent ops.
- <strong>config</strong> (<em>Optional</em><em>[</em>[*ConfigMapping*](config.mdx#dagster.ConfigMapping)<em>]</em>) – Defines the config of the graph, and how its schema maps to the config of its constituent ops.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Arbitrary metadata for any execution of the graph. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>. These tag values may be overwritten by tag values provided at invocation time.
- <strong>composition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>]</em>) – The function that defines this graph. Used to generate code references for this graph.
Examples:
```python
@op
def return_one():
return 1
@op
def add_one(num):
return num + 1
graph_def = GraphDefinition(
name='basic',
node_defs=[return_one, add_one],
dependencies={'add_one': {'num': DependencyDefinition('return_one')}},
)
```
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.alias'>alias <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L828' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.alias" class="hash-link"></a></Link></dt>
<dd>
Aliases the graph with a new name.
Can only be used in the context of a [`@graph`](#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.alias("my_graph_alias")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.execute_in_process'>execute_in_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L724' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.execute_in_process" class="hash-link"></a></Link></dt>
<dd>
Execute this graph in-process, collecting results in-memory.
Parameters:
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Run config to provide to execution. The configuration for the underlying graph should exist under the “ops” key.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The instance to execute against, an ephemeral one will be used if none provided.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources needed if any are required. Can provide resource instances directly, or resource definitions.
- <strong>raise_on_error</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to raise exceptions when they occur. Defaults to `True`.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of op selection queries (including single op names) to execute. For example: * `['some_op']`: selects `some_op` itself. * `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies). * `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down. * `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of the graph.
Returns: [`ExecuteInProcessResult`](execution.mdx#dagster.ExecuteInProcessResult)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.tag'>tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L843' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.tag" class="hash-link"></a></Link></dt>
<dd>
Attaches the provided tags to the graph immutably.
Can only be used in the context of a [`@graph`](#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.tag({"my_tag": "my_value"})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.to_job'>to_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L601' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.to_job" class="hash-link"></a></Link></dt>
<dd>
Make this graph in to an executable Job by providing remaining components required for execution.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name for the Job. Defaults to the name of the this graph.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em> [</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – Resources that are required by this graph for execution. If not defined, <cite>io_manager</cite> will default to filesystem.
- <strong>config</strong> –
Describes how the job is parameterized at runtime.
If no value is provided, then the schema for the job’s run config is a standard format based on its ops and resources.
If a dictionary is provided, then it must conform to the standard config schema, and it will be used as the job’s run config for the job whenever the job is executed. The values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
If a [`ConfigMapping`](config.mdx#dagster.ConfigMapping) object is provided, then the schema for the job’s run config is determined by the config mapping, and the ConfigMapping, which should return configuration in the standard format to configure the job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the job and can be used for searching and filtering in the UI. Values that are not already strings will be serialized as JSON. If <cite>run_tags</cite> is not set, then the content of <cite>tags</cite> will also be automatically appended to the tags of any runs of this job.
- <strong>run_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that will be automatically attached to runs launched by this job. Values that are not already strings will be serialized as JSON. These tag values may be overwritten by tag values provided at invocation time. If <cite>run_tags</cite> is set, then <cite>tags</cite> are not automatically appended to the tags of any runs of this job.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary information that will be attached to the JobDefinition and be viewable in the Dagster UI. Keys must be strings, and values must be python primitive types or one of the provided MetadataValue types
- <strong>logger_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – A dictionary of string logger identifiers to their implementations.
- <strong>executor_def</strong> (<em>Optional</em><em>[</em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em>) – How this Job will be executed. Defaults to [`multi_or_in_process_executor`](execution.mdx#dagster.multi_or_in_process_executor), which can be switched between multi-process and in-process modes of execution. The default mode of execution is multi-process.
- <strong>op_retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The default retry policy for all ops in this job. Only used if retry policy is not defined on the op definition or op invocation.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines a discrete set of partition keys that can parameterize the job. If this argument is supplied, the config argument can’t also be supplied.
- <strong>asset_layer</strong> (<em>Optional</em><em>[</em><em>AssetLayer</em><em>]</em>) – Top level information about the assets this job will produce. Generally should not be set manually.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of a job.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A sequence of strings identifying the owners of the job.
- <strong>automation_condition</strong> (<em>Optional</em><em>[</em>[*AutomationCondition*](assets.mdx#dagster.AutomationCondition)<em>]</em>) – An experimental, job-scoped automation condition. Only supported for asset jobs; generally set via <cite>define_asset_job</cite> rather than here.
Returns: JobDefinition
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.with_hooks'>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L858' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.with_hooks" class="hash-link"></a></Link></dt>
<dd>
Attaches the provided hooks to the graph immutably.
Can only be used in the context of a [`@graph`](#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.with_hooks({my_hook})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.with_retry_policy'>with_retry_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L873' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.with_retry_policy" class="hash-link"></a></Link></dt>
<dd>
Attaches the provided retry policy to the graph immutably.
Can only be used in the context of a [`@graph`](#dagster.graph), [`@job`](jobs.mdx#dagster.job), or `@asset_graph` decorated function.
<strong>Examples:</strong>
```python
@job
def do_it_all():
my_graph.with_retry_policy(RetryPolicy(max_retries=5))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.config_mapping'>`property` config_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.config_mapping" class="hash-link"></a></Link></dt>
<dd>
The config mapping for the graph, if present.
By specifying a config mapping function, you can override the configuration for the child nodes contained within a graph.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.input_mappings'>`property` input_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L413' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.input_mappings" class="hash-link"></a></Link></dt>
<dd>
Input mappings for the graph.
An input mapping is a mapping from an input of the graph to an input of a child node.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L809' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.name" class="hash-link"></a></Link></dt>
<dd>
The name of the graph.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.output_mappings'>`property` output_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L422' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.output_mappings" class="hash-link"></a></Link></dt>
<dd>
Output mappings for the graph.
An output mapping is a mapping from an output of the graph to an output of a child node.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphDefinition.tags'>`property` tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/graph_definition.py#L815' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphDefinition.tags" class="hash-link"></a></Link></dt>
<dd>
The tags associated with the graph.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphIn'>`class` dagster.GraphIn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/input.py#L484' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphIn" class="hash-link"></a></Link></dt>
<dd>
Represents information about an input that a graph maps.
Parameters: <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.GraphOut'>`class` dagster.GraphOut <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/output.py#L497' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.GraphOut" class="hash-link"></a></Link></dt>
<dd>
Represents information about the outputs that a graph maps.
Parameters: <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the output.
</dd>
</dl>
<div class="section" id="explicit-dependencies">
## Explicit dependencies
<dl>
<dt><Link class="anchor" id='dagster.DependencyDefinition'>`class` dagster.DependencyDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L627' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DependencyDefinition" class="hash-link"></a></Link></dt>
<dd>
Represents an edge in the DAG of nodes (ops or graphs) forming a job.
This object is used at the leaves of a dictionary structure that represents the complete
dependency structure of a job whose keys represent the dependent node and dependent
input, so this object only contains information about the dependee.
Concretely, if the input named ‘input’ of op_b depends on the output named ‘result’ of
op_a, and the output named ‘other_result’ of graph_a, the structure will look as follows:
```python
from dagster import DependencyDefinition
dependency_structure = {
'my_downstream_op': {
'input': DependencyDefinition('my_upstream_op', 'result')
},
'my_other_downstream_op': {
'input': DependencyDefinition('my_upstream_graph', 'result')
}
}
```
In general, users should prefer not to construct this class directly or use the
[`JobDefinition`](jobs.mdx#dagster.JobDefinition) API that requires instances of this class. Instead, use the
[`@job`](jobs.mdx#dagster.job) API:
```python
from dagster import job
@job
def the_job():
node_b(node_a())
```
Parameters:
- <strong>node</strong> (<em>str</em>) – The name of the node (op or graph) that is depended on, that is, from which the value passed between the two nodes originates.
- <strong>output</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output that is depended on. (default: “result”)
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of this dependency.
<dl>
<dt><Link class="anchor" id='dagster.DependencyDefinition.is_fan_in'>is_fan_in <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L692' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DependencyDefinition.is_fan_in" class="hash-link"></a></Link></dt>
<dd>
Return True if the dependency is fan-in (always False for DependencyDefinition).
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiDependencyDefinition'>`class` dagster.MultiDependencyDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L701' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiDependencyDefinition" class="hash-link"></a></Link></dt>
<dd>
Represents a fan-in edge in the DAG of op instances forming a job.
This object is used only when an input of type `List[T]` is assembled by fanning-in multiple
upstream outputs of type `T`.
This object is used at the leaves of a dictionary structure that represents the complete
dependency structure of a job whose keys represent the dependent ops or graphs and dependent
input, so this object only contains information about the dependee.
Concretely, if the input named ‘input’ of op_c depends on the outputs named ‘result’ of
op_a and op_b, this structure will look as follows:
```python
dependency_structure = {
'op_c': {
'input': MultiDependencyDefinition(
[
DependencyDefinition('op_a', 'result'),
DependencyDefinition('op_b', 'result')
]
)
}
}
```
In general, users should prefer not to construct this class directly or use the
[`JobDefinition`](jobs.mdx#dagster.JobDefinition) API that requires instances of this class. Instead, use the
[`@job`](jobs.mdx#dagster.job) API:
```python
@job
def the_job():
op_c(op_a(), op_b())
```
Parameters: <strong>dependencies</strong> (<em>List</em><em>[</em><em>Union</em><em>[</em>[*DependencyDefinition*](#dagster.DependencyDefinition)<em>, </em><em>Type</em><em>[</em><em>MappedInputPlaceHolder</em><em>]</em><em>]</em><em>]</em>) – List of
upstream dependencies fanned in to this input.
<dl>
<dt><Link class="anchor" id='dagster.MultiDependencyDefinition.get_dependencies_and_mappings'>get_dependencies_and_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L788' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiDependencyDefinition.get_dependencies_and_mappings" class="hash-link"></a></Link></dt>
<dd>
Return the combined list of dependencies contained by this object, inculding of [`DependencyDefinition`](#dagster.DependencyDefinition) and `MappedInputPlaceholder` objects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiDependencyDefinition.get_node_dependencies'>get_node_dependencies <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L778' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiDependencyDefinition.get_node_dependencies" class="hash-link"></a></Link></dt>
<dd>
Return the list of [`DependencyDefinition`](#dagster.DependencyDefinition) contained by this object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiDependencyDefinition.is_fan_in'>is_fan_in <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L783' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiDependencyDefinition.is_fan_in" class="hash-link"></a></Link></dt>
<dd>
Return <cite>True</cite> if the dependency is fan-in (always True for MultiDependencyDefinition).
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.NodeInvocation'>`class` dagster.NodeInvocation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dependency.py#L48' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.NodeInvocation" class="hash-link"></a></Link></dt>
<dd>
Identifies an instance of a node in a graph dependency structure.
Parameters:
- <strong>name</strong> (<em>str</em>) – Name of the node of which this is an instance.
- <strong>alias</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name specific to this instance of the node. Necessary when there are multiple instances of the same node.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Optional tags values to extend or override those set on the node definition.
- <strong>hook_defs</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*HookDefinition*](hooks.mdx#dagster.HookDefinition)<em>]</em><em>]</em>) – A set of hook definitions applied to the node instance.
Examples:
In general, users should prefer not to construct this class directly or use the
[`JobDefinition`](jobs.mdx#dagster.JobDefinition) API that requires instances of this class. Instead, use the
[`@job`](jobs.mdx#dagster.job) API:
```python
from dagster import job
@job
def my_job():
other_name = some_op.alias('other_name')
some_graph(other_name(some_op))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputMapping'>`class` dagster.OutputMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/output.py#L254' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputMapping" class="hash-link"></a></Link></dt>
<dd>
Defines an output mapping for a graph.
Parameters:
- <strong>graph_output_name</strong> (<em>str</em>) – Name of the output in the graph being mapped to.
- <strong>mapped_node_name</strong> (<em>str</em>) – Named of the node (op/graph) that the output is being mapped from.
- <strong>mapped_node_output_name</strong> (<em>str</em>) – Name of the output in the node (op/graph) that is being mapped from.
- <strong>graph_output_description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A description of the output in the graph being mapped from.
- <strong>from_dynamic_mapping</strong> (<em>bool</em>) – Set to true if the node being mapped to is a mapped dynamic node.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em>) – <span className="flag flag-warning">deprecated</span> The dagster type of the graph’s output being mapped to.
Examples:
```python
from dagster import OutputMapping, GraphDefinition, op, graph, GraphOut
@op
def emit_five(x):
return 5
# The following two graph definitions are equivalent
GraphDefinition(
name="the_graph",
node_defs=[emit_five],
output_mappings=[
OutputMapping(
graph_output_name="result", # Default output name
mapped_node_name="emit_five",
mapped_node_output_name="result"
)
]
)
@graph(out=GraphOut())
def the_graph():
return emit_five()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputMapping'>`class` dagster.InputMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/input.py#L326' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputMapping" class="hash-link"></a></Link></dt>
<dd>
Defines an input mapping for a graph.
Parameters:
- <strong>graph_input_name</strong> (<em>str</em>) – Name of the input in the graph being mapped from.
- <strong>mapped_node_name</strong> (<em>str</em>) – Named of the node (op/graph) that the input is being mapped to.
- <strong>mapped_node_input_name</strong> (<em>str</em>) – Name of the input in the node (op/graph) that is being mapped to.
- <strong>fan_in_index</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The index in to a fanned input, otherwise None.
- <strong>graph_input_description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A description of the input in the graph being mapped from.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em>) – DEPRECATED, to be removed in 2.0. The dagster type of the graph’s input being mapped from.
Examples:
```python
from dagster import InputMapping, GraphDefinition, op, graph
@op
def needs_input(x):
return x + 1
# The following two graph definitions are equivalent
GraphDefinition(
name="the_graph",
node_defs=[needs_input],
input_mappings=[
InputMapping(
graph_input_name="maps_x", mapped_node_name="needs_input",
mapped_node_input_name="x"
)
]
)
@graph
def the_graph(maps_x):
needs_input(maps_x)
```
</dd>
</dl>
</div></div>
---
---
title: 'hooks'
sidebar_position: 1000
title_meta: 'hooks API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'hooks Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="hooks">
# Hooks
<dl>
<dt><Link class="anchor" id='dagster.success_hook'>@dagster.success_hook <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/hook_decorator.py#L143' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.success_hook" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step success events with the specified parameters from the decorated function.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of this hook.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the hook.
Examples:
```python
@success_hook(required_resource_keys={'slack'})
def slack_message_on_success(context):
message = 'op {} succeeded'.format(context.op.name)
context.resources.slack.send_message(message)
@success_hook
def do_something_on_success(context):
do_something()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.failure_hook'>@dagster.failure_hook <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/hook_decorator.py#L216' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.failure_hook" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step failure events with the specified parameters from the decorated function.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of this hook.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the hook.
Examples:
```python
@failure_hook(required_resource_keys={'slack'})
def slack_message_on_failure(context):
message = 'op {} failed'.format(context.op.name)
context.resources.slack.send_message(message)
@failure_hook
def do_something_on_failure(context):
do_something()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookDefinition'>`class` dagster.HookDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/hook_definition.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookDefinition" class="hash-link"></a></Link></dt>
<dd>
Define a hook which can be triggered during a op execution (e.g. a callback on the step
execution failure event during a op execution).
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of this hook.
- <strong>hook_fn</strong> (<em>Callable</em>) – The callback function that will be triggered.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the hook.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext'>`class` dagster.HookContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object available to a hook function on an DagsterEvent.
<dl>
<dt><Link class="anchor" id='dagster.HookContext.hook_def'>`property` hook_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L75' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.hook_def" class="hash-link"></a></Link></dt>
<dd>
The hook that the context object belongs to.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L81' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.instance" class="hash-link"></a></Link></dt>
<dd>
The instance configured to run the current job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the job where this hook is being triggered.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L122' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.log" class="hash-link"></a></Link></dt>
<dd>
Centralized log dispatch from user code.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.op_config'>`property` op_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L110' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.op_config" class="hash-link"></a></Link></dt>
<dd>
The parsed config specific to this op.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.op_exception'>`property` op_exception <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L128' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.op_exception" class="hash-link"></a></Link></dt>
<dd>
The thrown exception in a failed op.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.op_output_metadata'>`property` op_output_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L168' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.op_output_metadata" class="hash-link"></a></Link></dt>
<dd>
The applied output metadata.
Returns a dictionary where keys are output names and the values are:
- the applied output metadata in the normal case
- a dictionary from mapping key to corresponding metadata in the mapped case
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.op_output_values'>`property` op_output_values <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L139' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.op_output_values" class="hash-link"></a></Link></dt>
<dd>
The computed output values.
Returns a dictionary where keys are output names and the values are:
- the output values in the normal case
- a dictionary from mapping key to corresponding value in the mapped case
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L98' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
Resources required by this hook.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L104' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.resources" class="hash-link"></a></Link></dt>
<dd>
Resources available in the hook context.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The id of the run where this hook is being triggered.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HookContext.step_key'>`property` step_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L92' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HookContext.step_key" class="hash-link"></a></Link></dt>
<dd>
The key for the step where this hook is being triggered.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_hook_context'>dagster.build_hook_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/hook.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_hook_context" class="hash-link"></a></Link></dt>
<dd>
Builds hook context from provided parameters.
`build_hook_context` can be used as either a function or a context manager. If there is a
provided resource to `build_hook_context` that is a context manager, then it must be used as a
context manager. This function can be used to provide the context argument to the invocation of
a hook definition.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to provide to the context. These can either be values or resource definitions.
- <strong>op</strong> (<em>Optional</em><em>[</em>[*OpDefinition*](ops.mdx#dagster.OpDefinition)<em>, </em><em>PendingNodeInvocation</em><em>]</em>) – The op definition which the hook may be associated with.
- <strong>run_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The id of the run in which the hook is invoked (provided for mocking purposes).
- <strong>job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the job in which the hook is used (provided for mocking purposes).
- <strong>op_exception</strong> (<em>Optional</em><em>[</em><em>Exception</em><em>]</em>) – The exception that caused the hook to be triggered.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The Dagster instance configured to run the hook.
Examples:
```python
context = build_hook_context()
hook_to_invoke(context)
with build_hook_context(resources={"foo": context_manager_resource}) as context:
hook_to_invoke(context)
```
</dd>
</dl>
</div>
---
---
title: 'internals'
sidebar_position: 1000
title_meta: 'internals API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'internals Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="internals">
# Internals
Note that APIs imported from Dagster submodules are not considered stable, and are potentially subject to change in the future.
If you find yourself consulting these docs because you are writing custom components and plug-ins,
please get in touch with the core team [on our Slack](https://join.slack.com/t/dagster/shared_invite/enQtNjEyNjkzNTA2OTkzLTI0MzdlNjU0ODVhZjQyOTMyMGM1ZDUwZDQ1YjJmYjI3YzExZGViMDI1ZDlkNTY5OThmYWVlOWM1MWVjN2I3NjU).
We’re curious what you’re up to, happy to help, excited for new community contributions, and eager
to make the system as easy to work with as possible – including for teams who are looking to
customize it.
<div class="section" id="executors">
## Executors
APIs for constructing custom executors. This is considered advanced usage. Please note that using Dagster-provided executors is considered stable, common usage.
<dl>
<dt><Link class="anchor" id='dagster.executor'>@dagster.executor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L212' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.executor" class="hash-link"></a></Link></dt>
<dd>
Define an executor.
The decorated function should accept an [`InitExecutorContext`](#dagster.InitExecutorContext) and return an instance
of [`Executor`](#dagster.Executor).
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the executor.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.executor_config</cite>. If not set, Dagster will accept any config provided for.
- <strong>requirements</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>ExecutorRequirement</em><em>]</em><em>]</em>) – Any requirements that must be met in order for the executor to be usable for a particular job execution.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecutorDefinition'>`class` dagster.ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecutorDefinition" class="hash-link"></a></Link></dt>
<dd>
An executor is responsible for executing the steps of a job.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the executor.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.executor_config</cite>. If not set, Dagster will accept any config provided.
- <strong>requirements</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>ExecutorRequirement</em><em>]</em><em>]</em>) – Any requirements that must be met in order for the executor to be usable for a particular job execution.
- <strong>executor_creation_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>]</em>) – Should accept an [`InitExecutorContext`](#dagster.InitExecutorContext) and return an instance of [`Executor`](#dagster.Executor)
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the executor.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A description of the executor.
<dl>
<dt><Link class="anchor" id='dagster.ExecutorDefinition.configured'>configured <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L159' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecutorDefinition.configured" class="hash-link"></a></Link></dt>
<dd>
Wraps this object in an object of the same type that provides configuration to the inner
object.
Using `configured` may result in config values being displayed in
the Dagster UI, so it is not recommended to use this API with sensitive values,
such as secrets.
Parameters:
- <strong>config_or_config_fn</strong> (<em>Union</em><em>[</em><em>Any</em><em>, </em><em>Callable</em><em>[</em><em>[</em><em>Any</em><em>]</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Either (1) Run configuration that fully satisfies this object’s config schema or (2) A function that accepts run configuration and returns run configuration that fully satisfies this object’s config schema. In the latter case, config_schema must be specified. When passing a function, it’s easiest to use [`configured()`](config.mdx#dagster.configured).
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the new definition. If not provided, the emitted definition will inherit the name of the <cite>ExecutorDefinition</cite> upon which this function is called.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – If config_or_config_fn is a function, the config schema that its input must satisfy. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Description of the new definition. If not specified, inherits the description of the definition being configured.
Returns (ConfigurableDefinition): A configured version of this object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecutorDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecutorDefinition.description" class="hash-link"></a></Link></dt>
<dd>
Description of executor, if provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecutorDefinition.executor_creation_fn'>`property` executor_creation_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L131' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecutorDefinition.executor_creation_fn" class="hash-link"></a></Link></dt>
<dd>
Callable that takes an [`InitExecutorContext`](#dagster.InitExecutorContext) and returns an instance of
[`Executor`](#dagster.Executor).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExecutorDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/executor_definition.py#L110' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExecutorDefinition.name" class="hash-link"></a></Link></dt>
<dd>
Name of the executor.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitExecutorContext'>`class` dagster.InitExecutorContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/executor/init.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitExecutorContext" class="hash-link"></a></Link></dt>
<dd>
Executor-specific initialization context.
Parameters:
- <strong>job</strong> (<em>IJob</em>) – The job to be executed.
- <strong>executor_def</strong> ([*ExecutorDefinition*](#dagster.ExecutorDefinition)) – The definition of the executor currently being constructed.
- <strong>executor_config</strong> (<em>dict</em>) – The parsed config passed to the executor.
- <strong>instance</strong> ([*DagsterInstance*](#dagster.DagsterInstance)) – The current instance.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Executor'>`class` dagster.Executor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/executor/base.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Executor" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster.Executor.execute'>`abstractmethod` execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/executor/base.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Executor.execute" class="hash-link"></a></Link></dt>
<dd>
For the given context and execution plan, orchestrate a series of sub plan executions in a way that satisfies the whole plan being executed.
Parameters:
- <strong>plan_context</strong> (<em>PlanOrchestrationContext</em>) – The plan’s orchestration context.
- <strong>execution_plan</strong> (<em>ExecutionPlan</em>) – The plan to execute.
Returns: A stream of dagster events.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Executor.retries'>`abstract` `property` retries <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/executor/base.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Executor.retries" class="hash-link"></a></Link></dt>
<dd>
Whether retries are enabled or disabled for this instance of the executor.
Executors should allow this to be controlled via configuration if possible.
Returns: RetryMode
</dd>
</dl>
</dd>
</dl>
<div class="section" id="file-manager">
### File manager
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager'>`class` dagster._core.storage.file_manager.FileManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager" class="hash-link"></a></Link></dt>
<dd>
Base class for all file managers in dagster.
The file manager is an interface that can be implemented by resources to provide abstract
access to a file system such as local disk, S3, or other cloud storage.
For examples of usage, see the documentation of the concrete file manager implementations.
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.copy_handle_to_local_temp'>`abstractmethod` copy_handle_to_local_temp <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L73' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.copy_handle_to_local_temp" class="hash-link"></a></Link></dt>
<dd>
Copy a file represented by a file handle to a temp file.
In an implementation built around an object store such as S3, this method would be expected
to download the file from S3 to local filesystem in a location assigned by the standard
library’s `python:tempfile` module.
Temp files returned by this method are <em>not</em> guaranteed to be reusable across solid
boundaries. For files that must be available across solid boundaries, use the
[`read()`](#dagster._core.storage.file_manager.FileManager.read),
[`read_data()`](#dagster._core.storage.file_manager.FileManager.read_data),
[`write()`](#dagster._core.storage.file_manager.FileManager.write), and
[`write_data()`](#dagster._core.storage.file_manager.FileManager.write_data) methods.
Parameters: <strong>file_handle</strong> ([*FileHandle*](#dagster.FileHandle)) – The handle to the file to make available as a local temp file.Returns: Path to the local temp file.Return type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.delete_local_temp'>`abstractmethod` delete_local_temp <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.delete_local_temp" class="hash-link"></a></Link></dt>
<dd>
Delete all local temporary files created by previous calls to
[`copy_handle_to_local_temp()`](#dagster._core.storage.file_manager.FileManager.copy_handle_to_local_temp).
Should typically only be called by framework implementors.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.read'>`abstractmethod` read <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L107' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.read" class="hash-link"></a></Link></dt>
<dd>
Return a file-like stream for the file handle.
This may incur an expensive network call for file managers backed by object stores
such as S3.
Parameters:
- <strong>file_handle</strong> ([*FileHandle*](#dagster.FileHandle)) – The file handle to make available as a stream.
- <strong>mode</strong> (<em>str</em>) – The mode in which to open the file. Default: `"rb"`.
Returns: A file-like stream.Return type: Union[TextIO, BinaryIO]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.read_data'>`abstractmethod` read_data <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L124' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.read_data" class="hash-link"></a></Link></dt>
<dd>
Return the bytes for a given file handle. This may incur an expensive network
call for file managers backed by object stores such as s3.
Parameters: <strong>file_handle</strong> ([*FileHandle*](#dagster.FileHandle)) – The file handle for which to return bytes.Returns: Bytes for a given file handle.Return type: bytes
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.write'>`abstractmethod` write <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L138' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.write" class="hash-link"></a></Link></dt>
<dd>
Write the bytes contained within the given file object into the file manager.
Parameters:
- <strong>file_obj</strong> (<em>Union</em><em>[</em><em>TextIO</em><em>, </em><em>StringIO</em><em>]</em>) – A file-like object.
- <strong>mode</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The mode in which to write the file into the file manager. Default: `"wb"`.
- <strong>ext</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – For file managers that support file extensions, the extension with which to write the file. Default: `None`.
Returns: A handle to the newly created file.Return type: [FileHandle](#dagster.FileHandle)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.file_manager.FileManager.write_data'>`abstractmethod` write_data <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L155' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.file_manager.FileManager.write_data" class="hash-link"></a></Link></dt>
<dd>
Write raw bytes into the file manager.
Parameters:
- <strong>data</strong> (<em>bytes</em>) – The bytes to write into the file manager.
- <strong>ext</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – For file managers that support file extensions, the extension with which to write the file. Default: `None`.
Returns: A handle to the newly created file.Return type: [FileHandle](#dagster.FileHandle)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.local_file_manager'>dagster.local_file_manager ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L171' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.local_file_manager" class="hash-link"></a></Link></dt>
<dd>
FileManager that provides abstract access to a local filesystem.
By default, files will be stored in <cite>\<local_artifact_storage>/storage/file_manager</cite> where
<cite>\<local_artifact_storage></cite> can be configured the `dagster.yaml` file in `$DAGSTER_HOME`.
Implements the [`FileManager`](#dagster._core.storage.file_manager.FileManager) API.
Examples:
```python
import tempfile
from dagster import job, local_file_manager, op
@op(required_resource_keys={"file_manager"})
def write_files(context):
fh_1 = context.resources.file_manager.write_data(b"foo")
with tempfile.NamedTemporaryFile("w+") as fd:
fd.write("bar")
fd.seek(0)
fh_2 = context.resources.file_manager.write(fd, mode="w", ext=".txt")
return (fh_1, fh_2)
@op(required_resource_keys={"file_manager"})
def read_files(context, file_handles):
fh_1, fh_2 = file_handles
assert context.resources.file_manager.read_data(fh_2) == b"bar"
fd = context.resources.file_manager.read(fh_2, mode="r")
assert fd.read() == "foo"
fd.close()
@job(resource_defs={"file_manager": local_file_manager})
def files_pipeline():
read_files(write_files())
```
Or to specify the file directory:
```python
@job(
resource_defs={
"file_manager": local_file_manager.configured({"base_dir": "/my/base/dir"})
}
)
def files_pipeline():
read_files(write_files())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FileHandle'>`class` dagster.FileHandle <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FileHandle" class="hash-link"></a></Link></dt>
<dd>
A reference to a file as manipulated by a FileManager.
Subclasses may handle files that are resident on the local file system, in an object store, or
in any arbitrary place where a file can be stored.
This exists to handle the very common case where you wish to write a computation that reads,
transforms, and writes files, but where you also want the same code to work in local development
as well as on a cluster where the files will be stored in a globally available object store
such as S3.
<dl>
<dt><Link class="anchor" id='dagster.FileHandle.path_desc'>`abstract` `property` path_desc <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L35' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FileHandle.path_desc" class="hash-link"></a></Link></dt>
<dd>
A representation of the file path for display purposes only.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LocalFileHandle'>`class` dagster.LocalFileHandle <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L43' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LocalFileHandle" class="hash-link"></a></Link></dt>
<dd>
A reference to a file on a local filesystem.
<dl>
<dt><Link class="anchor" id='dagster.LocalFileHandle.path'>`property` path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LocalFileHandle.path" class="hash-link"></a></Link></dt>
<dd>
The file’s path.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LocalFileHandle.path_desc'>`property` path_desc <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/file_manager.py#L56' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LocalFileHandle.path_desc" class="hash-link"></a></Link></dt>
<dd>
A representation of the file path for display purposes only.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="instance">
### Instance
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance'>`class` dagster.DagsterInstance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L68' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance" class="hash-link"></a></Link></dt>
<dd>
Core abstraction for managing Dagster’s access to storage and other resources.
Use DagsterInstance.get() to grab the current DagsterInstance which will load based on
the values in the `dagster.yaml` file in `$DAGSTER_HOME`.
Alternatively, DagsterInstance.ephemeral() can use used which provides a set of
transient in-memory components.
Configuration of this class should be done by setting values in `$DAGSTER_HOME/dagster.yaml`.
For example, to use Postgres for dagster storage, you can write a `dagster.yaml` such as the
following:
dagster.yaml
```YAML
storage:
postgres:
postgres_db:
username: my_username
password: my_password
hostname: my_hostname
db_name: my_database
port: 5432
```
Parameters:
- <strong>instance_type</strong> (<em>InstanceType</em>) – Indicates whether the instance is ephemeral or persistent. Users should not attempt to set this value directly or in their `dagster.yaml` files.
- <strong>local_artifact_storage</strong> ([*LocalArtifactStorage*](#dagster._core.storage.root.LocalArtifactStorage)) – The local artifact storage is used to configure storage for any artifacts that require a local disk, such as schedules, or when using the filesystem system storage to manage files and intermediates. By default, this will be a [`dagster._core.storage.root.LocalArtifactStorage`](#dagster._core.storage.root.LocalArtifactStorage). Configurable in `dagster.yaml` using the `ConfigurableClass` machinery.
- <strong>run_storage</strong> ([*RunStorage*](#dagster._core.storage.runs.RunStorage)) – The run storage is used to store metadata about ongoing and past pipeline runs. By default, this will be a [`dagster._core.storage.runs.SqliteRunStorage`](#dagster._core.storage.runs.SqliteRunStorage). Configurable in `dagster.yaml` using the `ConfigurableClass` machinery.
- <strong>event_storage</strong> ([*EventLogStorage*](#dagster._core.storage.event_log.EventLogStorage)) – Used to store the structured event logs generated by pipeline runs. By default, this will be a [`dagster._core.storage.event_log.SqliteEventLogStorage`](#dagster._core.storage.event_log.SqliteEventLogStorage). Configurable in `dagster.yaml` using the `ConfigurableClass` machinery.
- <strong>compute_log_manager</strong> (<em>Optional</em><em>[</em>[*ComputeLogManager*](#dagster._core.storage.compute_log_manager.ComputeLogManager)<em>]</em>) – The compute log manager handles stdout and stderr logging for op compute functions. By default, this will be a [`dagster._core.storage.local_compute_log_manager.LocalComputeLogManager`](#dagster._core.storage.local_compute_log_manager.LocalComputeLogManager). Configurable in `dagster.yaml` using the `ConfigurableClass` machinery.
- <strong>run_coordinator</strong> (<em>Optional</em><em>[</em><em>RunCoordinator</em><em>]</em>) – A runs coordinator may be used to manage the execution of pipeline runs.
- <strong>run_launcher</strong> (<em>Optional</em><em>[</em>[*RunLauncher*](#dagster._core.launcher.RunLauncher)<em>]</em>) – Optionally, a run launcher may be used to enable a Dagster instance to launch pipeline runs, e.g. on a remote Kubernetes cluster, in addition to running them locally.
- <strong>settings</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>]</em>) – Specifies certain per-instance settings, such as feature flags. These are set in the `dagster.yaml` under a set of whitelisted keys.
- <strong>ref</strong> (<em>Optional</em><em>[</em>[*InstanceRef*](#dagster._core.instance.InstanceRef)<em>]</em>) – Used by internal machinery to pass instances across process boundaries.
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.ephemeral'>`static` ephemeral <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L241' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.ephemeral" class="hash-link"></a></Link></dt>
<dd>
Create a <cite>DagsterInstance</cite> suitable for ephemeral execution, useful in test contexts. An
ephemeral instance uses mostly in-memory components. Use <cite>local_temp</cite> to create a test
instance that is fully persistent.
Parameters:
- <strong>tempdir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The path of a directory to be used for local artifact storage.
- <strong>preload</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>DebugRunPayload</em><em>]</em><em>]</em>) – A sequence of payloads to load into the instance’s run storage. Useful for debugging.
- <strong>settings</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>]</em>) – Settings for the instance.
Returns: An ephemeral DagsterInstance.Return type: [DagsterInstance](#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get'>`static` get <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L265' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get" class="hash-link"></a></Link></dt>
<dd>
Get the current <cite>DagsterInstance</cite> as specified by the `DAGSTER_HOME` environment variable.
Returns: The current DagsterInstance.Return type: [DagsterInstance](#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.local_temp'>`static` local_temp <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L277' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.local_temp" class="hash-link"></a></Link></dt>
<dd>
Create a DagsterInstance that uses a temporary directory for local storage. This is a
regular, fully persistent instance. Use <cite>ephemeral</cite> to get an ephemeral instance with
in-memory components.
Parameters:
- <strong>tempdir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The path of a directory to be used for local artifact storage.
- <strong>overrides</strong> (<em>Optional</em><em>[</em><em>DagsterInstanceOverrides</em><em>]</em>) – Override settings for the instance.
Returns: DagsterInstance
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.add_dynamic_partitions'>add_dynamic_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L602' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.add_dynamic_partitions" class="hash-link"></a></Link></dt>
<dd>
Add partitions to the specified [`DynamicPartitionsDefinition`](partitions.mdx#dagster.DynamicPartitionsDefinition) idempotently.
Does not add any partitions that already exist.
Parameters:
- <strong>partitions_def_name</strong> (<em>str</em>) – The name of the <cite>DynamicPartitionsDefinition</cite>.
- <strong>partition_keys</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – Partition keys to add.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.delete_dynamic_partition'>delete_dynamic_partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L616' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.delete_dynamic_partition" class="hash-link"></a></Link></dt>
<dd>
Delete a partition for the specified [`DynamicPartitionsDefinition`](partitions.mdx#dagster.DynamicPartitionsDefinition).
If the partition does not exist, exits silently.
Parameters:
- <strong>partitions_def_name</strong> (<em>str</em>) – The name of the <cite>DynamicPartitionsDefinition</cite>.
- <strong>partition_key</strong> (<em>str</em>) – Partition key to delete.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.delete_run'>delete_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L476' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.delete_run" class="hash-link"></a></Link></dt>
<dd>
Delete a run and all events generated by that from storage.
Parameters: <strong>run_id</strong> (<em>str</em>) – The id of the run to delete.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.fetch_materializations'>fetch_materializations <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L301' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.fetch_materializations" class="hash-link"></a></Link></dt>
<dd>
Return a list of materialization records stored in the event log storage.
Parameters:
- <strong>records_filter</strong> (<em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>AssetRecordsFilter</em><em>]</em>) – the filter by which to filter event records.
- <strong>limit</strong> (<em>int</em>) – Number of results to get.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Cursor to use for pagination. Defaults to None.
- <strong>ascending</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Sort the result in ascending order if True, descending otherwise. Defaults to descending.
Returns: Object containing a list of event log records and a cursor stringReturn type: EventRecordsResult
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.fetch_observations'>fetch_observations <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L325' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.fetch_observations" class="hash-link"></a></Link></dt>
<dd>
Return a list of observation records stored in the event log storage.
Parameters:
- <strong>records_filter</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>AssetRecordsFilter</em><em>]</em><em>]</em>) – the filter by which to filter event records.
- <strong>limit</strong> (<em>int</em>) – Number of results to get.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Cursor to use for pagination. Defaults to None.
- <strong>ascending</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Sort the result in ascending order if True, descending otherwise. Defaults to descending.
Returns: Object containing a list of event log records and a cursor stringReturn type: EventRecordsResult
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.fetch_run_status_changes'>fetch_run_status_changes <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L556' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.fetch_run_status_changes" class="hash-link"></a></Link></dt>
<dd>
Return a list of run_status_event records stored in the event log storage.
Parameters:
- <strong>records_filter</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*DagsterEventType*](execution.mdx#dagster.DagsterEventType)<em>, </em><em>RunStatusChangeRecordsFilter</em><em>]</em><em>]</em>) – the filter by which to filter event records.
- <strong>limit</strong> (<em>int</em>) – Number of results to get.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Cursor to use for pagination. Defaults to None.
- <strong>ascending</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Sort the result in ascending order if True, descending otherwise. Defaults to descending.
Returns: Object containing a list of event log records and a cursor stringReturn type: EventRecordsResult
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_asset_keys'>get_asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L349' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_asset_keys" class="hash-link"></a></Link></dt>
<dd>
Return a filtered subset of asset keys managed by this instance.
Parameters:
- <strong>prefix</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Return only assets having this key prefix.
- <strong>limit</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – Maximum number of keys to return.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Cursor to use for pagination.
Returns: List of asset keys.Return type: Sequence[[AssetKey](assets.mdx#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_asset_records'>get_asset_records <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L369' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_asset_records" class="hash-link"></a></Link></dt>
<dd>
Return an <cite>AssetRecord</cite> for each of the given asset keys.
Parameters: <strong>asset_keys</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – List of asset keys to retrieve records for.Returns: List of asset records.Return type: Sequence[[AssetRecord](#dagster._core.storage.event_log.AssetRecord)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_dynamic_partitions'>get_dynamic_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L628' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_dynamic_partitions" class="hash-link"></a></Link></dt>
<dd>
Get the set of partition keys for the specified [`DynamicPartitionsDefinition`](partitions.mdx#dagster.DynamicPartitionsDefinition).
Parameters: <strong>partitions_def_name</strong> (<em>str</em>) – The name of the <cite>DynamicPartitionsDefinition</cite>.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_latest_materialization_code_versions'>get_latest_materialization_code_versions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L384' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_latest_materialization_code_versions" class="hash-link"></a></Link></dt>
<dd>
Returns the code version used for the latest materialization of each of the provided
assets.
Parameters: <strong>asset_keys</strong> (<em>Iterable</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – The asset keys to find latest materialization code
versions for.Returns:
A dictionary with a key for each of the provided asset
keys. The values will be None if the asset has no materializations. If an asset does
not have a code version explicitly assigned to its definitions, but was
materialized, Dagster assigns the run ID as its code version.
Return type: Mapping[[AssetKey](assets.mdx#dagster.AssetKey), Optional[str]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_latest_materialization_event'>get_latest_materialization_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L403' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_latest_materialization_event" class="hash-link"></a></Link></dt>
<dd>
Fetch the latest materialization event for the given asset key.
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – Asset key to return materialization for.Returns:
The latest materialization event for the given asset
key, or <cite>None</cite> if the asset has not been materialized.
Return type: Optional[[EventLogEntry](#dagster.EventLogEntry)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_run_by_id'>get_run_by_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L487' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_run_by_id" class="hash-link"></a></Link></dt>
<dd>
Get a [`DagsterRun`](#dagster.DagsterRun) matching the provided <cite>run_id</cite>.
Parameters: <strong>run_id</strong> (<em>str</em>) – The id of the run to retrieve.Returns:
The run corresponding to the given id. If no run matching the id
is found, return <cite>None</cite>.
Return type: Optional[[DagsterRun](#dagster.DagsterRun)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_run_record_by_id'>get_run_record_by_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L503' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_run_record_by_id" class="hash-link"></a></Link></dt>
<dd>
Get a `RunRecord` matching the provided <cite>run_id</cite>.
Parameters: <strong>run_id</strong> (<em>str</em>) – The id of the run record to retrieve.Returns:
The run record corresponding to the given id. If no run matching
the id is found, return <cite>None</cite>.
Return type: Optional[[RunRecord](#dagster._core.storage.dagster_run.RunRecord)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_run_records'>get_run_records <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L522' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_run_records" class="hash-link"></a></Link></dt>
<dd>
Return a list of run records stored in the run storage, sorted by the given column in given order.
Parameters:
- <strong>filters</strong> (<em>Optional</em><em>[</em>[*RunsFilter*](#dagster.RunsFilter)<em>]</em>) – the filter by which to filter runs.
- <strong>limit</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – Number of results to get. Defaults to infinite.
- <strong>order_by</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the column to sort by. Defaults to id.
- <strong>ascending</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Sort the result in ascending order if True, descending otherwise. Defaults to descending.
Returns: List of run records stored in the run storage.Return type: List[[RunRecord](#dagster._core.storage.dagster_run.RunRecord)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.get_status_by_partition'>get_status_by_partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L417' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.get_status_by_partition" class="hash-link"></a></Link></dt>
<dd>
Get the current status of provided partition_keys for the provided asset.
Parameters:
- <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – The asset to get per-partition status for.
- <strong>partition_keys</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – The partitions to get status for.
- <strong>partitions_def</strong> ([*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)) – The PartitionsDefinition of the asset to get per-partition status for.
Returns: status for each partition keyReturn type: Optional[Mapping[str, AssetPartitionStatus]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.has_asset_key'>has_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L439' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.has_asset_key" class="hash-link"></a></Link></dt>
<dd>
Return true if this instance manages the given asset key.
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – Asset key to check.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.has_dynamic_partition'>has_dynamic_partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L650' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.has_dynamic_partition" class="hash-link"></a></Link></dt>
<dd>
Check if a partition key exists for the [`DynamicPartitionsDefinition`](partitions.mdx#dagster.DynamicPartitionsDefinition).
Parameters:
- <strong>partitions_def_name</strong> (<em>str</em>) – The name of the <cite>DynamicPartitionsDefinition</cite>.
- <strong>partition_key</strong> (<em>str</em>) – Partition key to check.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.report_runless_asset_event'>report_runless_asset_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.report_runless_asset_event" class="hash-link"></a></Link></dt>
<dd>
Record an event log entry related to assets that does not belong to a Dagster run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterInstance.wipe_assets'>wipe_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/instance.py#L463' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterInstance.wipe_assets" class="hash-link"></a></Link></dt>
<dd>
Wipes asset event history from the event log for the given asset keys.
Parameters: <strong>asset_keys</strong> (<em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – Asset keys to wipe.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.instance.InstanceRef'>`class` dagster._core.instance.InstanceRef <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/instance/ref.py#L192' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.instance.InstanceRef" class="hash-link"></a></Link></dt>
<dd>
Serializable representation of a `DagsterInstance`.
Users should not instantiate this class directly.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._serdes.ConfigurableClass'>`class` dagster._serdes.ConfigurableClass <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_serdes/config_class.py#L123' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._serdes.ConfigurableClass" class="hash-link"></a></Link></dt>
<dd>
Abstract mixin for classes that can be loaded from config.
This supports a powerful plugin pattern which avoids both a) a lengthy, hard-to-synchronize list
of conditional imports / optional extras_requires in dagster core and b) a magic directory or
file in which third parties can place plugin packages. Instead, the intention is to make, e.g.,
run storage, pluggable with a config chunk like:
```yaml
run_storage:
module: very_cool_package.run_storage
class: SplendidRunStorage
config:
magic_word: "quux"
```
This same pattern should eventually be viable for other system components, e.g. engines.
The `ConfigurableClass` mixin provides the necessary hooks for classes to be instantiated from
an instance of `ConfigurableClassData`.
Pieces of the Dagster system which we wish to make pluggable in this way should consume a config
type such as:
```python
{'module': str, 'class': str, 'config': Field(Permissive())}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._serdes.ConfigurableClassData'>`class` dagster._serdes.ConfigurableClassData <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_serdes/config_class.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._serdes.ConfigurableClassData" class="hash-link"></a></Link></dt>
<dd>
Serializable tuple describing where to find a class and the config fragment that should
be used to instantiate it.
Users should not instantiate this class directly.
Classes intended to be serialized in this way should implement the
`dagster.serdes.ConfigurableClass` mixin.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.root.LocalArtifactStorage'>`class` dagster._core.storage.root.LocalArtifactStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/root.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.root.LocalArtifactStorage" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
<div class="section" id="storage">
### Storage
<dl>
<dt><Link class="anchor" id='dagster._core.storage.base_storage.DagsterStorage'>`class` dagster._core.storage.base_storage.DagsterStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/base_storage.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.base_storage.DagsterStorage" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for Dagster persistent storage, for reading and writing data for runs,
events, and schedule/sensor state.
Users should not directly instantiate concrete subclasses of this class; they are instantiated
by internal machinery when `dagster-webserver` and `dagster-daemon` load, based on the values in the
`dagster.yaml` file in `$DAGSTER_HOME`. Configuration of concrete subclasses of this class
should be done by setting values in that file.
</dd>
</dl>
</div>
<div class="section" id="run-storage">
### Run storage
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun'>`class` dagster.DagsterRun <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L219' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun" class="hash-link"></a></Link></dt>
<dd>
Serializable internal representation of a dagster run, as stored in a
[`RunStorage`](#dagster._core.storage.runs.RunStorage).
Parameters:
- <strong>job_name</strong> (<em>str</em>) – The name of the job executed in this run.
- <strong>run_id</strong> (<em>str</em>) – The ID of the run.
- <strong>run_config</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em>) – The config for the run.
- <strong>asset_selection</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – The assets selected for this run.
- <strong>asset_check_selection</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em>[*AssetCheckKey*](asset-checks.mdx#dagster.AssetCheckKey)<em>]</em><em>]</em>) – The asset checks selected for this run.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The op queries provided by the user.
- <strong>resolved_op_selection</strong> (<em>Optional</em><em>[</em><em>AbstractSet</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The resolved set of op names to execute.
- <strong>step_keys_to_execute</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The step keys to execute.
- <strong>status</strong> ([*DagsterRunStatus*](#dagster.DagsterRunStatus)) – The status of the run.
- <strong>tags</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em>) – The tags applied to the run.
- <strong>root_run_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the root run in the run’s group.
- <strong>parent_run_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the parent run in the run’s group.
- <strong>job_snapshot_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the job snapshot.
- <strong>execution_plan_snapshot_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the execution plan snapshot.
- <strong>remote_job_origin</strong> (<em>Optional</em><em>[</em><em>RemoteJobOrigin</em><em>]</em>) – The origin of the executed job.
- <strong>job_code_origin</strong> (<em>Optional</em><em>[</em><em>JobPythonOrigin</em><em>]</em>) – The origin of the job code.
- <strong>has_repository_load_data</strong> (<em>bool</em>) – Whether the run has repository load data.
- <strong>run_op_concurrency</strong> (<em>Optional</em><em>[</em><em>RunOpConcurrency</em><em>]</em>) – The op concurrency information for the run.
- <strong>partitions_subset</strong> (<em>Optional</em><em>[</em><em>PartitionsSubset</em><em>]</em>) – The subset of partitions to execute.
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_cancelable'>`property` is_cancelable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L525' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_cancelable" class="hash-link"></a></Link></dt>
<dd>
If this run an be canceled.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_failure'>`property` is_failure <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L537' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_failure" class="hash-link"></a></Link></dt>
<dd>
If this run has failed.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_failure_or_canceled'>`property` is_failure_or_canceled <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L543' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_failure_or_canceled" class="hash-link"></a></Link></dt>
<dd>
If this run has either failed or was canceled.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_finished'>`property` is_finished <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L519' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_finished" class="hash-link"></a></Link></dt>
<dd>
If this run has completely finished execution.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_resume_retry'>`property` is_resume_retry <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L549' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_resume_retry" class="hash-link"></a></Link></dt>
<dd>
If this run was created from retrying another run from the point of failure.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRun.is_success'>`property` is_success <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L531' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRun.is_success" class="hash-link"></a></Link></dt>
<dd>
If this run has successfully finished executing.
Type: bool
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus'>`class` dagster.DagsterRunStatus <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L53' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRunStatus" class="hash-link"></a></Link></dt>
<dd>
The status of run execution.
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.CANCELED'>CANCELED `=` 'CANCELED'<a href="#dagster.DagsterRunStatus.CANCELED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.CANCELING'>CANCELING `=` 'CANCELING'<a href="#dagster.DagsterRunStatus.CANCELING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.FAILURE'>FAILURE `=` 'FAILURE'<a href="#dagster.DagsterRunStatus.FAILURE" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.MANAGED'>MANAGED `=` 'MANAGED'<a href="#dagster.DagsterRunStatus.MANAGED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.NOT_STARTED'>NOT_STARTED `=` 'NOT_STARTED'<a href="#dagster.DagsterRunStatus.NOT_STARTED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.QUEUED'>QUEUED `=` 'QUEUED'<a href="#dagster.DagsterRunStatus.QUEUED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.STARTED'>STARTED `=` 'STARTED'<a href="#dagster.DagsterRunStatus.STARTED" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.STARTING'>STARTING `=` 'STARTING'<a href="#dagster.DagsterRunStatus.STARTING" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunStatus.SUCCESS'>SUCCESS `=` 'SUCCESS'<a href="#dagster.DagsterRunStatus.SUCCESS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunsFilter'>`class` dagster.RunsFilter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L610' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunsFilter" class="hash-link"></a></Link></dt>
<dd>
Defines a filter across job runs, for use when querying storage directly.
Each field of the RunsFilter represents a logical AND with each other. For
example, if you specify job_name and tags, then you will receive only runs
with the specified job_name AND the specified tags. If left blank, then
all values will be permitted for that field.
Parameters:
- <strong>run_ids</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of job run_id values.
- <strong>job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the job to query for. If blank, all job_names will be accepted.
- <strong>statuses</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em>[*DagsterRunStatus*](#dagster.DagsterRunStatus)<em>]</em><em>]</em>) – A list of run statuses to filter by. If blank, all run statuses will be allowed.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – A dictionary of run tags to query by. All tags specified here must be present for a given run to pass the filter.
- <strong>snapshot_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the job snapshot to query for. Intended for internal use.
- <strong>updated_after</strong> (<em>Optional</em><em>[</em><em>DateTime</em><em>]</em>) – Filter by runs that were last updated before this datetime.
- <strong>created_before</strong> (<em>Optional</em><em>[</em><em>DateTime</em><em>]</em>) – Filter by runs that were created before this datetime.
- <strong>exclude_subruns</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If true, runs that were launched to backfill historical data will be excluded from results.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.runs.RunStorage'>`class` dagster._core.storage.runs.RunStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/runs/base.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.runs.RunStorage" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for storing pipeline run history.
Note that run storages using SQL databases as backing stores should implement
[`SqlRunStorage`](#dagster._core.storage.runs.SqlRunStorage).
Users should not directly instantiate concrete subclasses of this class; they are instantiated
by internal machinery when `dagster-webserver` and `dagster-graphql` load, based on the values in the
`dagster.yaml` file in `$DAGSTER_HOME`. Configuration of concrete subclasses of this class
should be done by setting values in that file.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.runs.SqlRunStorage'>`class` dagster._core.storage.runs.SqlRunStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/runs/sql_run_storage.py#L90' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.runs.SqlRunStorage" class="hash-link"></a></Link></dt>
<dd>
Base class for SQL based run storages.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.runs.SqliteRunStorage'>`class` dagster._core.storage.runs.SqliteRunStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/runs/sqlite/sqlite_run_storage.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.runs.SqliteRunStorage" class="hash-link"></a></Link></dt>
<dd>
SQLite-backed run storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
This is the default run storage when none is specified in the `dagster.yaml`.
To explicitly specify SQLite for run storage, you can add a block such as the following to your
`dagster.yaml`:
```YAML
run_storage:
module: dagster._core.storage.runs
class: SqliteRunStorage
config:
base_dir: /path/to/dir
```
The `base_dir` param tells the run storage where on disk to store the database.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.dagster_run.RunRecord'>`class` dagster._core.storage.dagster_run.RunRecord <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/dagster_run.py#L702' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.dagster_run.RunRecord" class="hash-link"></a></Link></dt>
<dd>
Internal representation of a run record, as stored in a
[`RunStorage`](#dagster._core.storage.runs.RunStorage).
Users should not invoke this class directly.
</dd>
</dl>
See also: [`dagster_postgres.PostgresRunStorage`](../../integrations/libraries/postgres/dagster-postgres.mdx#dagster_postgres.PostgresRunStorage) and [`dagster_mysql.MySQLRunStorage`](../../integrations/libraries/mysql/dagster-mysql.mdx#dagster_mysql.MySQLRunStorage).
</div>
<div class="section" id="event-log-storage">
### Event log storage
<dl>
<dt><Link class="anchor" id='dagster.EventLogEntry'>`class` dagster.EventLogEntry <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/log.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogEntry" class="hash-link"></a></Link></dt>
<dd>
Entries in the event log.
Users should not instantiate this object directly. These entries may originate from the logging machinery (DagsterLogManager/context.log), from
framework events (e.g. EngineEvent), or they may correspond to events yielded by user code
(e.g. Output).
Parameters:
- <strong>error_info</strong> (<em>Optional</em><em>[</em><em>SerializableErrorInfo</em><em>]</em>) – Error info for an associated exception, if any, as generated by serializable_error_info_from_exc_info and friends.
- <strong>level</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>int</em><em>]</em>) – The Python log level at which to log this event. Note that framework and user code events are also logged to Python logging. This value may be an integer or a (case-insensitive) string member of PYTHON_LOGGING_LEVELS_NAMES.
- <strong>user_message</strong> (<em>str</em>) – For log messages, this is the user-generated message.
- <strong>run_id</strong> (<em>str</em>) – The id of the run which generated this event.
- <strong>timestamp</strong> (<em>float</em>) – The Unix timestamp of this event.
- <strong>step_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The step key for the step which generated this event. Some events are generated outside of a step context.
- <strong>job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The job which generated this event. Some events are generated outside of a job context.
- <strong>dagster_event</strong> (<em>Optional</em><em>[</em>[*DagsterEvent*](execution.mdx#dagster.DagsterEvent)<em>]</em>) – For framework and user events, the associated structured event.
<dl>
<dt><Link class="anchor" id='dagster.EventLogEntry.get_dagster_event'>get_dagster_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/log.py#L98' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogEntry.get_dagster_event" class="hash-link"></a></Link></dt>
<dd>
DagsterEvent: Returns the DagsterEvent contained within this entry. If this entry does not
contain a DagsterEvent, an error will be raised.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EventLogEntry.dagster_event_type'>`property` dagster_event_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/log.py#L117' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogEntry.dagster_event_type" class="hash-link"></a></Link></dt>
<dd>
The type of the DagsterEvent contained by this entry, if any.
Type: Optional[[DagsterEventType](execution.mdx#dagster.DagsterEventType)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EventLogEntry.is_dagster_event'>`property` is_dagster_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/log.py#L92' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogEntry.is_dagster_event" class="hash-link"></a></Link></dt>
<dd>
If this entry contains a DagsterEvent.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EventLogEntry.message'>`property` message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/events/log.py#L123' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogEntry.message" class="hash-link"></a></Link></dt>
<dd>
Return the message from the structured DagsterEvent if present, fallback to user_message.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EventLogRecord'>`class` dagster.EventLogRecord <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/event_api.py#L96' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventLogRecord" class="hash-link"></a></Link></dt>
<dd>
Internal representation of an event record, as stored in a
[`EventLogStorage`](#dagster._core.storage.event_log.EventLogStorage).
Users should not instantiate this class directly.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.EventRecordsFilter'>`class` dagster.EventRecordsFilter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/event_api.py#L162' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EventRecordsFilter" class="hash-link"></a></Link></dt>
<dd>
Defines a set of filter fields for fetching a set of event log entries or event log records.
Parameters:
- <strong>event_type</strong> ([*DagsterEventType*](execution.mdx#dagster.DagsterEventType)) – Filter argument for dagster event type
- <strong>asset_key</strong> (<em>Optional</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – Asset key for which to get asset materialization event entries / records.
- <strong>asset_partitions</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Filter parameter such that only asset events with a partition value matching one of the provided values. Only valid when the <cite>asset_key</cite> parameter is provided.
- <strong>after_cursor</strong> (<em>Optional</em><em>[</em><em>EventCursor</em><em>]</em>) – Filter parameter such that only records with storage_id greater than the provided value are returned. Using a run-sharded events cursor will result in a significant performance gain when run against a SqliteEventLogStorage implementation (which is run-sharded)
- <strong>before_cursor</strong> (<em>Optional</em><em>[</em><em>EventCursor</em><em>]</em>) – Filter parameter such that records with storage_id less than the provided value are returned. Using a run-sharded events cursor will result in a significant performance gain when run against a SqliteEventLogStorage implementation (which is run-sharded)
- <strong>after_timestamp</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – Filter parameter such that only event records for events with timestamp greater than the provided value are returned.
- <strong>before_timestamp</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – Filter parameter such that only event records for events with timestamp less than the provided value are returned.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunShardedEventsCursor'>`class` dagster.RunShardedEventsCursor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/event_api.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunShardedEventsCursor" class="hash-link"></a></Link></dt>
<dd>
Pairs an id-based event log cursor with a timestamp-based run cursor, for improved
performance on run-sharded event log storages (e.g. the default SqliteEventLogStorage). For
run-sharded storages, the id field is ignored, since they may not be unique across shards.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.event_log.EventLogStorage'>`class` dagster._core.storage.event_log.EventLogStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/event_log/base.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.event_log.EventLogStorage" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for storing structured event logs from pipeline runs.
Note that event log storages using SQL databases as backing stores should implement
[`SqlEventLogStorage`](#dagster._core.storage.event_log.SqlEventLogStorage).
Users should not directly instantiate concrete subclasses of this class; they are instantiated
by internal machinery when `dagster-webserver` and `dagster-graphql` load, based on the values in the
`dagster.yaml` file in `$DAGSTER_HOME`. Configuration of concrete subclasses of this class
should be done by setting values in that file.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.event_log.SqlEventLogStorage'>`class` dagster._core.storage.event_log.SqlEventLogStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/event_log/sql_event_log.py#L149' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.event_log.SqlEventLogStorage" class="hash-link"></a></Link></dt>
<dd>
Base class for SQL backed event log storages.
Distinguishes between run-based connections and index connections in order to support run-level
sharding, while maintaining the ability to do cross-run queries
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.event_log.SqliteEventLogStorage'>`class` dagster._core.storage.event_log.SqliteEventLogStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/event_log/sqlite/sqlite_event_log.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.event_log.SqliteEventLogStorage" class="hash-link"></a></Link></dt>
<dd>
SQLite-backed event log storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file insqliteve
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
This is the default event log storage when none is specified in the `dagster.yaml`.
To explicitly specify SQLite for event log storage, you can add a block such as the following
to your `dagster.yaml`:
```YAML
event_log_storage:
module: dagster._core.storage.event_log
class: SqliteEventLogStorage
config:
base_dir: /path/to/dir
```
The `base_dir` param tells the event log storage where on disk to store the databases. To
improve concurrent performance, event logs are stored in a separate SQLite database for each
run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.event_log.ConsolidatedSqliteEventLogStorage'>`class` dagster._core.storage.event_log.ConsolidatedSqliteEventLogStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/event_log/sqlite/consolidated_sqlite_event_log.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.event_log.ConsolidatedSqliteEventLogStorage" class="hash-link"></a></Link></dt>
<dd>
SQLite-backed consolidated event log storage intended for test cases only.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
To explicitly specify the consolidated SQLite for event log storage, you can add a block such as
the following to your `dagster.yaml`:
```YAML
run_storage:
module: dagster._core.storage.event_log
class: ConsolidatedSqliteEventLogStorage
config:
base_dir: /path/to/dir
```
The `base_dir` param tells the event log storage where on disk to store the database.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.event_log.AssetRecord'>`class` dagster._core.storage.event_log.AssetRecord <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/event_log/base.py#L133' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.event_log.AssetRecord" class="hash-link"></a></Link></dt>
<dd>
Internal representation of an asset record, as stored in a [`EventLogStorage`](#dagster._core.storage.event_log.EventLogStorage).
Users should not invoke this class directly.
</dd>
</dl>
See also: [`dagster_postgres.PostgresEventLogStorage`](../../integrations/libraries/postgres/dagster-postgres.mdx#dagster_postgres.PostgresEventLogStorage) and [`dagster_mysql.MySQLEventLogStorage`](../../integrations/libraries/mysql/dagster-mysql.mdx#dagster_mysql.MySQLEventLogStorage).
</div>
<div class="section" id="compute-log-manager">
### Compute log manager
<dl>
<dt><Link class="anchor" id='dagster._core.storage.compute_log_manager.ComputeLogManager'>`class` dagster._core.storage.compute_log_manager.ComputeLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/compute_log_manager.py#L184' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.compute_log_manager.ComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for capturing the unstructured logs (stdout/stderr) in the current
process, stored / retrieved with a provided log_key.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.local_compute_log_manager.LocalComputeLogManager'>`class` dagster._core.storage.local_compute_log_manager.LocalComputeLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/local_compute_log_manager.py#L45' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.local_compute_log_manager.LocalComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
Stores copies of stdout & stderr for each compute step locally on disk.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.noop_compute_log_manager.NoOpComputeLogManager'>`class` dagster._core.storage.noop_compute_log_manager.NoOpComputeLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/noop_compute_log_manager.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.noop_compute_log_manager.NoOpComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
When enabled for a Dagster instance, stdout and stderr will not be available for any step.
</dd>
</dl>
See also: `dagster_aws.S3ComputeLogManager`.
</div>
<div class="section" id="run-launcher">
### Run launcher
<dl>
<dt><Link class="anchor" id='dagster._core.launcher.RunLauncher'>`class` dagster._core.launcher.RunLauncher <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/launcher/base.py#L59' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.launcher.RunLauncher" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.launcher.DefaultRunLauncher'>`class` dagster._core.launcher.DefaultRunLauncher <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/launcher/default_run_launcher.py#L29' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.launcher.DefaultRunLauncher" class="hash-link"></a></Link></dt>
<dd>
Launches runs against running GRPC servers.
</dd>
</dl>
</div>
<div class="section" id="run-coordinator">
### Run coordinator
<dl>
<dt><Link class="anchor" id='dagster._core.run_coordinator.DefaultRunCoordinator'>dagster._core.run_coordinator.DefaultRunCoordinator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/run_coordinator/sync_in_memory_run_coordinator.py#L13' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.run_coordinator.DefaultRunCoordinator" class="hash-link"></a></Link></dt>
<dd>
alias of `SyncInMemoryRunCoordinator`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.run_coordinator.QueuedRunCoordinator'>dagster._core.run_coordinator.QueuedRunCoordinator RunCoordinator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/run_coordinator/queued_run_coordinator.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.run_coordinator.QueuedRunCoordinator" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Enqueues runs via the run storage, to be deqeueued by the Dagster Daemon process. Requires
the Dagster Daemon process to be alive in order for runs to be launched.
</dd>
</dl>
</div>
<div class="section" id="scheduling">
### Scheduling
<dl>
<dt><Link class="anchor" id='dagster._core.scheduler.Scheduler'>`class` dagster._core.scheduler.Scheduler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/scheduler/scheduler.py#L64' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.scheduler.Scheduler" class="hash-link"></a></Link></dt>
<dd>
Abstract base class for a scheduler. This component is responsible for interfacing with
an external system such as cron to ensure scheduled repeated execution according.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.schedules.ScheduleStorage'>`class` dagster._core.storage.schedules.ScheduleStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/schedules/base.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.schedules.ScheduleStorage" class="hash-link"></a></Link></dt>
<dd>
Abstract class for managing persistance of scheduler artifacts.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.schedules.SqlScheduleStorage'>`class` dagster._core.storage.schedules.SqlScheduleStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/schedules/sql_schedule_storage.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.schedules.SqlScheduleStorage" class="hash-link"></a></Link></dt>
<dd>
Base class for SQL backed schedule storage.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.storage.schedules.SqliteScheduleStorage'>`class` dagster._core.storage.schedules.SqliteScheduleStorage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/schedules/sqlite/sqlite_schedule_storage.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.storage.schedules.SqliteScheduleStorage" class="hash-link"></a></Link></dt>
<dd>
Local SQLite backed schedule storage.
</dd>
</dl>
See also: [`dagster_postgres.PostgresScheduleStorage`](../../integrations/libraries/postgres/dagster-postgres.mdx#dagster_postgres.PostgresScheduleStorage) and [`dagster_mysql.MySQLScheduleStorage`](../../integrations/libraries/mysql/dagster-mysql.mdx#dagster_mysql.MySQLScheduleStorage).
</div>
<div class="section" id="exception-handling">
### Exception handling
<dl>
<dt><Link class="anchor" id='dagster._core.errors.user_code_error_boundary'>dagster._core.errors.user_code_error_boundary <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/errors.py#L245' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.errors.user_code_error_boundary" class="hash-link"></a></Link></dt>
<dd>
Wraps the execution of user-space code in an error boundary. This places a uniform
policy around any user code invoked by the framework. This ensures that all user
errors are wrapped in an exception derived from DagsterUserCodeExecutionError,
and that the original stack trace of the user error is preserved, so that it
can be reported without confusing framework code in the stack trace, if a
tool author wishes to do so.
Examples:
.. code-block:: python
>
with user_code_error_boundary(
# Pass a class that inherits from DagsterUserCodeExecutionError
DagsterExecutionStepExecutionError,
# Pass a function that produces a message
“Error occurred during step execution”
):
call_user_provided_function()
</dd>
</dl>
</div>
<div class="section" id="step-launchers-superseded">
### Step Launchers (Superseded)
Learn how to migrate from Step Launchers to Dagster Pipes in the [migration guide](https://docs.dagster.io/integrations/external-pipelines/migrating-from-step-launchers-to-pipes).
<dl>
<dt><Link class="anchor" id='dagster.StepLauncher'>`class` dagster.StepLauncher <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/step_launcher.py#L67' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StepLauncher" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
While there is no plan to remove this functionality, for new projects, we recommend using Dagster Pipes. For more information, see https://docs.dagster.io/integrations/external-pipelines.
:::
A StepLauncher is responsible for executing steps, either in-process or in an external process.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StepRunRef'>`class` dagster.StepRunRef <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/step_launcher.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StepRunRef" class="hash-link"></a></Link></dt>
<dd>
A serializable object that specifies what’s needed to hydrate a step so
that it can be executed in a process outside the plan process.
Users should not instantiate this class directly.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StepExecutionContext'>`class` dagster.StepExecutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L416' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StepExecutionContext" class="hash-link"></a></Link></dt>
<dd>
Context for the execution of a step. Users should not instantiate this class directly.
This context assumes that user code can be run directly, and thus includes resource and information.
</dd>
</dl>
</div></div></div>
---
---
title: 'io managers'
sidebar_position: 1000
title_meta: 'io managers API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'io managers Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="io-managers">
# IO Managers
IO managers are user-provided objects that store op outputs and load them as inputs to downstream
ops.
<dl>
<dt><Link class="anchor" id='dagster.ConfigurableIOManager'>`class` dagster.ConfigurableIOManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/pythonic_config/io_manager.py#L204' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ConfigurableIOManager" class="hash-link"></a></Link></dt>
<dd>
Base class for Dagster IO managers that utilize structured config.
This class is a subclass of both [`IOManagerDefinition`](#dagster.IOManagerDefinition), [`Config`](config.mdx#dagster.Config),
and [`IOManager`](#dagster.IOManager). Implementers must provide an implementation of the
`handle_output()` and `load_input()` methods.
Example definition:
```python
class MyIOManager(ConfigurableIOManager):
path_prefix: List[str]
def _get_path(self, context) -> str:
return "/".join(context.asset_key.path)
def handle_output(self, context, obj):
write_csv(self._get_path(context), obj)
def load_input(self, context):
return read_csv(self._get_path(context))
Definitions(
...,
resources={
"io_manager": MyIOManager(path_prefix=["my", "prefix"])
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ConfigurableIOManagerFactory'>`class` dagster.ConfigurableIOManagerFactory <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/pythonic_config/io_manager.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ConfigurableIOManagerFactory" class="hash-link"></a></Link></dt>
<dd>
Base class for Dagster IO managers that utilize structured config. This base class
is useful for cases in which the returned IO manager is not the same as the class itself
(e.g. when it is a wrapper around the actual IO manager implementation).
This class is a subclass of both [`IOManagerDefinition`](#dagster.IOManagerDefinition) and [`Config`](config.mdx#dagster.Config).
Implementers should provide an implementation of the `resource_function()` method,
which should return an instance of [`IOManager`](#dagster.IOManager).
Example definition:
```python
class ExternalIOManager(IOManager):
def __init__(self, connection):
self._connection = connection
def handle_output(self, context, obj):
...
def load_input(self, context):
...
class ConfigurableExternalIOManager(ConfigurableIOManagerFactory):
username: str
password: str
def create_io_manager(self, context) -> IOManager:
with database.connect(username, password) as connection:
return MyExternalIOManager(connection)
Definitions(
...,
resources={
"io_manager": ConfigurableExternalIOManager(
username="dagster",
password=EnvVar("DB_PASSWORD")
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IOManager'>`class` dagster.IOManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/io_manager.py#L129' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IOManager" class="hash-link"></a></Link></dt>
<dd>
Base class for user-provided IO managers.
IOManagers are used to store op outputs and load them as inputs to downstream ops.
Extend this class to handle how objects are loaded and stored. Users should implement
`handle_output` to store an object and `load_input` to retrieve an object.
<dl>
<dt><Link class="anchor" id='dagster.IOManager.handle_output'>`abstractmethod` handle_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/io_manager.py#L152' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IOManager.handle_output" class="hash-link"></a></Link></dt>
<dd>
User-defined method that stores an output of an op.
Parameters:
- <strong>context</strong> ([*OutputContext*](#dagster.OutputContext)) – The context of the step output that produces this object.
- <strong>obj</strong> (<em>Any</em>) – The object, returned by the op, to be stored.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IOManager.load_input'>`abstractmethod` load_input <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/io_manager.py#L139' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IOManager.load_input" class="hash-link"></a></Link></dt>
<dd>
User-defined method that loads an input to an op.
Parameters: <strong>context</strong> ([*InputContext*](#dagster.InputContext)) – The input context, which describes the input that’s being loaded
and the upstream output that’s being loaded from.Returns: The data object.Return type: Any
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IOManagerDefinition'>`class` dagster.IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L247' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IOManagerDefinition" class="hash-link"></a></Link></dt>
<dd>
Definition of an IO manager resource.
IOManagers are used to store op outputs and load them as inputs to downstream ops.
An IOManagerDefinition is a [`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition) whose <cite>resource_fn</cite> returns an
[`IOManager`](#dagster.IOManager).
The easiest way to create an IOManagerDefnition is with the [`@io_manager`](#dagster.io_manager)
decorator.
<dl>
<dt><Link class="anchor" id='dagster.IOManagerDefinition.hardcoded_io_manager'>`static` hardcoded_io_manager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/io_manager.py#L111' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IOManagerDefinition.hardcoded_io_manager" class="hash-link"></a></Link></dt>
<dd>
A helper function that creates an `IOManagerDefinition` with a hardcoded IOManager.
Parameters:
- <strong>value</strong> ([*IOManager*](#dagster.IOManager)) – A hardcoded IO Manager which helps mock the definition.
- <strong>description</strong> (<em>[</em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The description of the IO Manager. Defaults to None.
Returns: A hardcoded resource.Return type: [[IOManagerDefinition](#dagster.IOManagerDefinition)]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.io_manager'>@dagster.io_manager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/io_manager.py#L178' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.io_manager" class="hash-link"></a></Link></dt>
<dd>
Define an IO manager.
IOManagers are used to store op outputs and load them as inputs to downstream ops.
The decorated function should accept an [`InitResourceContext`](resources.mdx#dagster.InitResourceContext) and return an
[`IOManager`](#dagster.IOManager).
Parameters:
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the resource config. Configuration data available in <cite>init_context.resource_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the resource.
- <strong>output_config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for per-output config. If not set, no per-output configuration will be allowed.
- <strong>input_config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for per-input config. If not set, Dagster will accept any config provided.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the object manager.
- <strong>version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The version of a resource function. Two wrapped resource functions should only have the same version if they produce the same resource definition when provided with the same inputs.
<strong>Examples:</strong>
```python
class MyIOManager(IOManager):
def handle_output(self, context, obj):
write_csv("some/path")
def load_input(self, context):
return read_csv("some/path")
@io_manager
def my_io_manager(init_context):
return MyIOManager()
@op(out=Out(io_manager_key="my_io_manager_key"))
def my_op(_):
return do_stuff()
@job(resource_defs={"my_io_manager_key": my_io_manager})
def my_job():
my_op()
```
</dd>
</dl>
<div class="section" id="input-and-output-contexts">
## Input and output contexts
<dl>
<dt><Link class="anchor" id='dagster.InputContext'>`class` dagster.InputContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L32' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object available to the load_input method of [`InputManager`](#dagster.InputManager).
Users should not instantiate this object directly. In order to construct
an <cite>InputContext</cite> for testing an IO Manager’s <cite>load_input</cite> method, use
[`dagster.build_input_context()`](#dagster.build_input_context).
Example:
```python
from dagster import IOManager, InputContext
class MyIOManager(IOManager):
def load_input(self, context: InputContext):
...
```
<dl>
<dt><Link class="anchor" id='dagster.InputContext.get_asset_identifier'>get_asset_identifier <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L460' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.get_asset_identifier" class="hash-link"></a></Link></dt>
<dd>
The sequence of strings making up the AssetKey for the asset being loaded as an input.
If the asset is partitioned, the identifier contains the partition key as the final element in the
sequence. For example, for the asset key `AssetKey(["foo", "bar", "baz"])`, materialized with
partition key “2023-06-01”, `get_asset_identifier` will return `["foo", "bar", "baz", "2023-06-01"]`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.get_identifier'>get_identifier <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L433' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.get_identifier" class="hash-link"></a></Link></dt>
<dd>
Utility method to get a collection of identifiers that as a whole represent a unique
step input.
If not using memoization, the unique identifier collection consists of
- `run_id`: the id of the run which generates the input.
- `step_key`: the key for a compute step.
- `name`: the name of the output. (default: ‘result’).
If using memoization, the `version` corresponding to the step output is used in place of
the `run_id`.
Returns: A list of identifiers, i.e. (run_id or version), step_key, and output_nameReturn type: List[str, …]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L274' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_key" class="hash-link"></a></Link></dt>
<dd>
The `AssetKey` of the asset that is being loaded as an input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_partition_key'>`property` asset_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L339' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for input asset.
Raises an error if the input asset has no partitioning, or if the run covers a partition
range for the input asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_partition_key_range'>`property` asset_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L361' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The partition key range for input asset.
Raises an error if the input asset has no partitioning.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_partition_keys'>`property` asset_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L387' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_partition_keys" class="hash-link"></a></Link></dt>
<dd>
The partition keys for input asset.
Raises an error if the input asset has no partitioning.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_partitions_def'>`property` asset_partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_partitions_def" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition on the upstream asset corresponding to this input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.asset_partitions_time_window'>`property` asset_partitions_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L401' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.asset_partitions_time_window" class="hash-link"></a></Link></dt>
<dd>
The time window for the partitions of the input asset.
Raises an error if either of the following are true:
- The input asset has no partitioning.
- The input asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.config'>`property` config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L180' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.config" class="hash-link"></a></Link></dt>
<dd>
The config attached to the input that we’re loading.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.dagster_type'>`property` dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L209' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.dagster_type" class="hash-link"></a></Link></dt>
<dd>
The type of this input.
Dagster types do not propagate from an upstream output to downstream inputs,
and this property only captures type information for the input that is either
passed in explicitly with [`AssetIn`](assets.mdx#dagster.AssetIn) or [`In`](ops.mdx#dagster.In), or can be
infered from type hints. For an asset input, the Dagster type from the upstream
asset definition is ignored.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.definition_metadata'>`property` definition_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L193' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.definition_metadata" class="hash-link"></a></Link></dt>
<dd>
A dict of metadata that is assigned to the InputDefinition that we’re loading.
This property only contains metadata passed in explicitly with [`AssetIn`](assets.mdx#dagster.AssetIn)
or [`In`](ops.mdx#dagster.In). To access metadata of an upstream asset or op definition,
use the definition_metadata in [`InputContext.upstream_output`](#dagster.InputContext.upstream_output).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.has_asset_key'>`property` has_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L266' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.has_asset_key" class="hash-link"></a></Link></dt>
<dd>
Returns True if an asset is being loaded as input, otherwise returns False. A return value of False
indicates that an output from an op is being loaded as the input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.has_asset_partitions'>`property` has_asset_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L333' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.has_asset_partitions" class="hash-link"></a></Link></dt>
<dd>
Returns True if the asset being loaded as input is partitioned.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.has_input_name'>`property` has_input_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L139' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.has_input_name" class="hash-link"></a></Link></dt>
<dd>
If we’re the InputContext is being used to load the result of a run from outside the run,
then it won’t have an input name.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.has_partition_key'>`property` has_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L313' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.has_partition_key" class="hash-link"></a></Link></dt>
<dd>
Whether the current run is a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L227' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.log" class="hash-link"></a></Link></dt>
<dd>
The log manager to use for this input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.metadata'>`property` metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L186' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.metadata" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.0.
Use definition_metadata instead.
:::
Use definitiion_metadata instead.
Type: Deprecated
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L147' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.name" class="hash-link"></a></Link></dt>
<dd>
The name of the input that we’re loading.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L168' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The definition of the op that’s loading the input.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L319' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run.
Raises an error if the current run is not a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.resource_config'>`property` resource_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L239' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.resource_config" class="hash-link"></a></Link></dt>
<dd>
The config associated with the resource that initializes the InputManager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L245' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.resources" class="hash-link"></a></Link></dt>
<dd>
The resources required by the resource that initializes the
input manager. If using the `@input_manager()` decorator, these resources
correspond to those requested with the <cite>required_resource_keys</cite> parameter.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputContext.upstream_output'>`property` upstream_output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L203' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputContext.upstream_output" class="hash-link"></a></Link></dt>
<dd>
Info about the output that produced the object we’re loading.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext'>`class` dagster.OutputContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L60' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext" class="hash-link"></a></Link></dt>
<dd>
The context object that is available to the <cite>handle_output</cite> method of an [`IOManager`](#dagster.IOManager).
Users should not instantiate this object directly. To construct an
<cite>OutputContext</cite> for testing an IO Manager’s <cite>handle_output</cite> method, use
[`dagster.build_output_context()`](#dagster.build_output_context).
Example:
```python
from dagster import IOManager, OutputContext
class MyIOManager(IOManager):
def handle_output(self, context: OutputContext, obj):
...
```
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.add_output_metadata'>add_output_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L765' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.add_output_metadata" class="hash-link"></a></Link></dt>
<dd>
Add a dictionary of metadata to the handled output.
Metadata entries added will show up in the HANDLED_OUTPUT and ASSET_MATERIALIZATION events for the run.
Parameters: <strong>metadata</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em>) – A metadata dictionary to log
Examples:
```python
from dagster import IOManager
class MyIOManager(IOManager):
def handle_output(self, context, obj):
context.add_output_metadata({"foo": "bar"})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.get_asset_identifier'>get_asset_identifier <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L674' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.get_asset_identifier" class="hash-link"></a></Link></dt>
<dd>
The sequence of strings making up the AssetKey for the asset being stored as an output.
If the asset is partitioned, the identifier contains the partition key as the final element in the
sequence. For example, for the asset key `AssetKey(["foo", "bar", "baz"])` materialized with
partition key “2023-06-01”, `get_asset_identifier` will return `["foo", "bar", "baz", "2023-06-01"]`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.get_identifier'>get_identifier <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L628' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.get_identifier" class="hash-link"></a></Link></dt>
<dd>
Utility method to get a collection of identifiers that as a whole represent a unique
step output.
If not using memoization, the unique identifier collection consists of
- `run_id`: the id of the run which generates the output.
- `step_key`: the key for a compute step.
- `name`: the name of the output. (default: ‘result’).
If using memoization, the `version` corresponding to the step output is used in place of
the `run_id`.
Returns: A list of identifiers, i.e. (run_id or version), step_key, and output_nameReturn type: Sequence[str, …]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.log_event'>log_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L697' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.log_event" class="hash-link"></a></Link></dt>
<dd>
Log an AssetMaterialization or AssetObservation from within the body of an io manager’s <cite>handle_output</cite> method.
Events logged with this method will appear in the event log.
Parameters: <strong>event</strong> (<em>Union</em><em>[</em>[*AssetMaterialization*](ops.mdx#dagster.AssetMaterialization)<em>, </em>[*AssetObservation*](assets.mdx#dagster.AssetObservation)<em>]</em>) – The event to log.
Examples:
```python
from dagster import IOManager, AssetMaterialization
class MyIOManager(IOManager):
def handle_output(self, context, obj):
context.log_event(AssetMaterialization("foo"))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L358' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_key" class="hash-link"></a></Link></dt>
<dd>
The `AssetKey` of the asset that is being stored as an output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_partition_key'>`property` asset_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L467' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for output asset.
Raises an error if the output asset has no partitioning, or if the run covers a partition
range for the output asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_partition_key_range'>`property` asset_partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L496' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The partition key range for output asset.
Raises an error if the output asset has no partitioning.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_partition_keys'>`property` asset_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L528' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_partition_keys" class="hash-link"></a></Link></dt>
<dd>
The partition keys for the output asset.
Raises an error if the output asset has no partitioning.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_partitions_def'>`property` asset_partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L370' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_partitions_def" class="hash-link"></a></Link></dt>
<dd>
The PartitionsDefinition on the asset corresponding to this output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_partitions_time_window'>`property` asset_partitions_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L548' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_partitions_time_window" class="hash-link"></a></Link></dt>
<dd>
The time window for the partitions of the output asset.
Raises an error if either of the following are true:
- The output asset has no partitioning.
- The output asset is not partitioned with a TimeWindowPartitionsDefinition or a
MultiPartitionsDefinition with one time-partitioned dimension.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.asset_spec'>`property` asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L388' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.asset_spec" class="hash-link"></a></Link></dt>
<dd>
The `AssetSpec` that is being stored as an output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.config'>`property` config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L276' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.config" class="hash-link"></a></Link></dt>
<dd>
The configuration for the output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.dagster_type'>`property` dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L294' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.dagster_type" class="hash-link"></a></Link></dt>
<dd>
The type of this output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.definition_metadata'>`property` definition_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L247' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.definition_metadata" class="hash-link"></a></Link></dt>
<dd>
A dict of the metadata that is assigned to the OutputDefinition that produced
the output. Metadata is assigned to an OutputDefinition either directly on the OutputDefinition
or in the @asset decorator.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.has_asset_key'>`property` has_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L350' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.has_asset_key" class="hash-link"></a></Link></dt>
<dd>
Returns True if an asset is being stored, otherwise returns False. A return value of False
indicates that an output from an op is being stored.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.has_asset_partitions'>`property` has_asset_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L453' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.has_asset_partitions" class="hash-link"></a></Link></dt>
<dd>
Returns True if the asset being stored is partitioned.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.has_partition_key'>`property` has_partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L417' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.has_partition_key" class="hash-link"></a></Link></dt>
<dd>
Whether the current run is a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L306' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.log" class="hash-link"></a></Link></dt>
<dd>
The log manager to use for this output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.mapping_key'>`property` mapping_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L270' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.mapping_key" class="hash-link"></a></Link></dt>
<dd>
The key that identifies a unique mapped output. None for regular outputs.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.metadata'>`property` metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L240' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.metadata" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.0.
Use definition_metadata instead.
:::
used definition_metadata instead.
Type: Deprecated
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L206' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.name" class="hash-link"></a></Link></dt>
<dd>
The name of the output that produced the output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L282' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The definition of the op that produced the output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.output_metadata'>`property` output_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L256' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.output_metadata" class="hash-link"></a></Link></dt>
<dd>
A dict of the metadata that is assigned to the output at execution time.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the current run.
Raises an error if the current run is not a partitioned run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.resource_config'>`property` resource_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L324' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.resource_config" class="hash-link"></a></Link></dt>
<dd>
The config associated with the resource that initializes the InputManager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L330' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.resources" class="hash-link"></a></Link></dt>
<dd>
The resources required by the output manager, specified by the <cite>required_resource_keys</cite>
parameter.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L228' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The id of the run that produced the output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.step_key'>`property` step_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L194' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.step_key" class="hash-link"></a></Link></dt>
<dd>
The step_key for the compute step that produced the output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OutputContext.version'>`property` version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L318' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OutputContext.version" class="hash-link"></a></Link></dt>
<dd>
The version of the output.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_input_context'>dagster.build_input_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/input.py#L555' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_input_context" class="hash-link"></a></Link></dt>
<dd>
Builds input context from provided parameters.
`build_input_context` can be used as either a function, or a context manager. If resources
that are also context managers are provided, then `build_input_context` must be used as a
context manager.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the input that we’re loading.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The config attached to the input that we’re loading.
- <strong>definition_metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of metadata that is assigned to the InputDefinition that we’re loading for.
- <strong>upstream_output</strong> (<em>Optional</em><em>[</em>[*OutputContext*](#dagster.OutputContext)<em>]</em>) – Info about the output that produced the object we’re loading.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em>) – The type of this input.
- <strong>resource_config</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resource config to make available from the input context. This usually corresponds to the config provided to the resource that loads the input manager.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to make available from the context. For a given key, you can provide either an actual instance of an object, or a resource definition.
- <strong>asset_key</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em><em>]</em>) – The asset key attached to the InputDefinition.
- <strong>op_def</strong> (<em>Optional</em><em>[</em>[*OpDefinition*](ops.mdx#dagster.OpDefinition)<em>]</em>) – The definition of the op that’s loading the input.
- <strong>step_context</strong> (<em>Optional</em><em>[</em>[*StepExecutionContext*](internals.mdx#dagster.StepExecutionContext)<em>]</em>) – For internal use.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – String value representing partition key to execute with.
- <strong>asset_partition_key_range</strong> (<em>Optional</em><em>[</em>[*PartitionKeyRange*](partitions.mdx#dagster.PartitionKeyRange)<em>]</em>) – The range of asset partition keys to load.
- <strong>asset_partitions_def</strong> – Optional[PartitionsDefinition]: The PartitionsDefinition of the asset being loaded.
Examples:
```python
build_input_context()
with build_input_context(resources={"foo": context_manager_resource}) as context:
do_something
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_output_context'>dagster.build_output_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/output.py#L916' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_output_context" class="hash-link"></a></Link></dt>
<dd>
Builds output context from provided parameters.
`build_output_context` can be used as either a function, or a context manager. If resources
that are also context managers are provided, then `build_output_context` must be used as a
context manager.
Parameters:
- <strong>step_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The step_key for the compute step that produced the output.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the output that produced the output.
- <strong>definition_metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of the metadata that is assigned to the OutputDefinition that produced the output.
- <strong>mapping_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The key that identifies a unique mapped output. None for regular outputs.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The configuration for the output.
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em>) – The type of this output.
- <strong>version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The version of the output.
- <strong>resource_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resource config to make available from the input context. This usually corresponds to the config provided to the resource that loads the output manager.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Resources</em><em>]</em>) – The resources to make available from the context. For a given key, you can provide either an actual instance of an object, or a resource definition.
- <strong>op_def</strong> (<em>Optional</em><em>[</em>[*OpDefinition*](ops.mdx#dagster.OpDefinition)<em>]</em>) – The definition of the op that produced the output.
- <strong>asset_key</strong> – Optional[Union[AssetKey, Sequence[str], str]]: The asset key corresponding to the output.
- <strong>partition_key</strong> – Optional[str]: String value representing partition key to execute with.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> Deprecated. Use definition_metadata instead.
- <strong>output_metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of the metadata that is assigned to the output at execution time.
Examples:
```python
build_output_context()
with build_output_context(resources={"foo": context_manager_resource}) as context:
do_something
```
</dd>
</dl>
</div>
<div class="section" id="built-in-io-managers">
## Built-in IO managers
<dl>
<dt><Link class="anchor" id='dagster.FilesystemIOManager'>dagster.FilesystemIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/fs_io_manager.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FilesystemIOManager" class="hash-link"></a></Link></dt>
<dd>
Built-in filesystem IO manager that stores and retrieves values using pickling.
The base directory that the pickle files live inside is determined by:
- The IO manager’s “base_dir” configuration value, if specified. Otherwise…
- A “storage/” directory underneath the value for “local_artifact_storage” in your dagster.yaml file, if specified. Otherwise…
- A “storage/” directory underneath the directory that the DAGSTER_HOME environment variable points to, if that environment variable is specified. Otherwise…
- A temporary directory.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
So, with a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
1. Attach an IO manager to a set of assets using the reserved resource key `"io_manager"`.
```python
from dagster import Definitions, asset, FilesystemIOManager
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": FilesystemIOManager(base_dir="/my/base/path")
},
)
```
2. Specify a job-level IO manager using the reserved resource key `"io_manager"`,
which will set the given IO manager on all ops in a job.
```python
from dagster import FilesystemIOManager, job, op
@op
def op_a():
# create df ...
return df
@op
def op_b(df):
return df[:5]
@job(
resource_defs={
"io_manager": FilesystemIOManager(base_dir="/my/base/path")
}
)
def job():
op_b(op_a())
```
3. Specify IO manager on [`Out`](ops.mdx#dagster.Out), which allows you to set different IO managers on
different step outputs.
```python
from dagster import FilesystemIOManager, job, op, Out
@op(out=Out(io_manager_key="my_io_manager"))
def op_a():
# create df ...
return df
@op
def op_b(df):
return df[:5]
@job(resource_defs={"my_io_manager": FilesystemIOManager()})
def job():
op_b(op_a())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InMemoryIOManager'>dagster.InMemoryIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/mem_io_manager.py#L6' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InMemoryIOManager" class="hash-link"></a></Link></dt>
<dd>
I/O manager that stores and retrieves values in memory. After execution is complete, the values will
be garbage-collected. Note that this means that each run will not have access to values from previous runs.
</dd>
</dl>
The `UPathIOManager` can be used to easily define filesystem-based IO Managers.
<dl>
<dt><Link class="anchor" id='dagster.UPathIOManager'>`class` dagster.UPathIOManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/upath_io_manager.py#L56' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.UPathIOManager" class="hash-link"></a></Link></dt>
<dd>
Abstract IOManager base class compatible with local and cloud storage via <cite>universal-pathlib</cite> and <cite>fsspec</cite>.
Features:
- handles partitioned assets
- handles loading a single upstream partition
- handles loading multiple upstream partitions (with respect to [`PartitionMapping`](partitions.mdx#dagster.PartitionMapping))
- supports loading multiple partitions concurrently with async <cite>load_from_path</cite> method
- the <cite>get_metadata</cite> method can be customized to add additional metadata to the output
- the <cite>allow_missing_partitions</cite> metadata value can be set to <cite>True</cite> to skip missing partitions (the default behavior is to raise an error)
</dd>
</dl>
</div>
<div class="section" id="io-storage-migration">
## IO storage migration
<dl>
<dt><Link class="anchor" id='dagster.migrate_io_storage'>dagster.migrate_io_storage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/migrate.py#L59' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.migrate_io_storage" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Migrate asset data from one IO manager to another.
Reads all materialized assets from a source IO manager and writes them to the
destination IO manager using `load_input` / `handle_output`. This is useful
when switching between IO managers (e.g., S3 to GCS, filesystem to S3) without
re-materializing assets.
The set of assets to iterate over and the source IO manager for each asset can be
provided in one of two ways:
- <strong>Via a ``Definitions`` object</strong> (the `definitions` parameter): The assets and their IO manager configuration are resolved from the `Definitions` object. An `instance` must also be provided to discover materialized partitions.
- <strong>Via an ``OpExecutionContext``</strong> (the `context` parameter): The assets and their IO manager configuration are resolved from the code location that the currently executing op belongs to. The `DagsterInstance` is obtained from the context automatically.
Exactly one of `definitions` or `context` must be provided.
Parameters:
- <strong>definitions</strong> (<em>Optional</em><em>[</em>[*Definitions*](definitions.mdx#dagster.Definitions)<em>]</em>) – A Definitions object containing asset definitions and their currently-configured IO manager resources (the source). Each asset’s `io_manager_key` is used to resolve the IO manager for loading. Mutually exclusive with `context`.
- <strong>destination_io_manager</strong> ([*IOManager*](#dagster.IOManager)) – The IO manager to write asset data to.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – A DagsterInstance used to discover materialized assets and partitions. Required when using `definitions`, ignored when using `context` (the instance is obtained from the context).
- <strong>context</strong> (<em>Optional</em><em>[</em>[*OpExecutionContext*](execution.mdx#dagster.OpExecutionContext)<em>]</em>) – An execution context from within an op or asset. When provided, the assets and IO managers are resolved from the code location, and the instance is obtained from the context. Mutually exclusive with `definitions`.
- <strong>selection</strong> (<em>Optional</em><em>[</em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em>) – An optional asset selection to filter which assets to migrate. Defaults to all assets.
- <strong>should_skip</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – An optional callback that receives an asset key and partition key (None for unpartitioned assets) and returns True if the asset should be skipped.
- <strong>batch_partitions</strong> (<em>bool</em>) – If True, migrates partitioned assets in batches using `PartitionKeyRange` contexts instead of one partition at a time. The batch size is determined by the asset’s `BackfillPolicy.max_partitions_per_run` (or all partitions at once if no BackfillPolicy is set). Both the source and destination IO managers must support multi-partition contexts. Defaults to False.
- <strong>transform</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>object</em><em>]</em><em>, </em><em>object</em><em>]</em><em>]</em>) – An optional function to transform each loaded value before storing it in the destination IO manager. Useful when the in-memory Python type differs between source and destination IO managers.
Returns: EntitySubsets of migrated, skipped, and failed assets.Return type: [MigrateIOStorageResult](#dagster.MigrateIOStorageResult)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MigrateIOStorageResult'>`class` dagster.MigrateIOStorageResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/migrate.py#L42' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MigrateIOStorageResult" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Result of a call to [`migrate_io_storage()`](#dagster.migrate_io_storage).
Parameters:
- <strong>migrated</strong> – EntitySubsets of assets/partitions that were successfully migrated.
- <strong>skipped</strong> – EntitySubsets of assets/partitions that were skipped (via `should_skip`).
- <strong>failed</strong> – EntitySubsets of assets/partitions that failed to migrate.
</dd>
</dl>
</div>
<div class="section" id="input-managers">
## Input managers
Input managers load inputs from either upstream outputs or from provided default values.
<dl>
<dt><Link class="anchor" id='dagster.input_manager'>@dagster.input_manager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/input_manager.py#L122' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.input_manager" class="hash-link"></a></Link></dt>
<dd>
Define an input manager.
Input managers load op inputs, either from upstream outputs or by providing default values.
The decorated function should accept a [`InputContext`](#dagster.InputContext) and resource config, and return
a loaded object that will be passed into one of the inputs of an op.
The decorator produces an [`InputManagerDefinition`](#dagster.InputManagerDefinition).
Parameters:
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the resource-level config. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the resource.
- <strong>input_config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – A schema for the input-level config. Each input that uses this input manager can be configured separately using this config. If not set, Dagster will accept any config provided.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by the input manager.
- <strong>version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The version of the input manager definition.
<strong>Examples:</strong>
```python
from dagster import input_manager, op, job, In
@input_manager
def csv_loader(_):
return read_csv("some/path")
@op(ins={"input1": In(input_manager_key="csv_loader_key")})
def my_op(_, input1):
do_stuff(input1)
@job(resource_defs={"csv_loader_key": csv_loader})
def my_job():
my_op()
@input_manager(config_schema={"base_dir": str})
def csv_loader(context):
return read_csv(context.resource_config["base_dir"] + "/some/path")
@input_manager(input_config_schema={"path": str})
def csv_loader(context):
return read_csv(context.config["path"])
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputManager'>`class` dagster.InputManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/input_manager.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputManager" class="hash-link"></a></Link></dt>
<dd>
Base interface for classes that are responsible for loading solid inputs.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InputManagerDefinition'>`class` dagster.InputManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L247' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InputManagerDefinition" class="hash-link"></a></Link></dt>
<dd>
Definition of an input manager resource.
Input managers load op inputs.
An InputManagerDefinition is a [`ResourceDefinition`](resources.mdx#dagster.ResourceDefinition) whose resource_fn returns an
[`InputManager`](#dagster.InputManager).
The easiest way to create an InputManagerDefinition is with the
[`@input_manager`](#dagster.input_manager) decorator.
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster.fs_io_manager'>dagster.fs_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/fs_io_manager.py#L140' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.fs_io_manager" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use FilesystemIOManager directly instead.
:::
Built-in filesystem IO manager that stores and retrieves values using pickling.
The base directory that the pickle files live inside is determined by:
- The IO manager’s “base_dir” configuration value, if specified. Otherwise…
- A “storage/” directory underneath the value for “local_artifact_storage” in your dagster.yaml file, if specified. Otherwise…
- A “storage/” directory underneath the directory that the DAGSTER_HOME environment variable points to, if that environment variable is specified. Otherwise…
- A temporary directory.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
So, with a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
1. Attach an IO manager to a set of assets using the reserved resource key `"io_manager"`.
```python
from dagster import Definitions, asset, fs_io_manager
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": fs_io_manager.configured({"base_dir": "/my/base/path"})
},
)
```
2. Specify a job-level IO manager using the reserved resource key `"io_manager"`,
which will set the given IO manager on all ops in a job.
```python
from dagster import fs_io_manager, job, op
@op
def op_a():
# create df ...
return df
@op
def op_b(df):
return df[:5]
@job(
resource_defs={
"io_manager": fs_io_manager.configured({"base_dir": "/my/base/path"})
}
)
def job():
op_b(op_a())
```
3. Specify IO manager on [`Out`](ops.mdx#dagster.Out), which allows you to set different IO managers on
different step outputs.
```python
from dagster import fs_io_manager, job, op, Out
@op(out=Out(io_manager_key="my_io_manager"))
def op_a():
# create df ...
return df
@op
def op_b(df):
return df[:5]
@job(resource_defs={"my_io_manager": fs_io_manager})
def job():
op_b(op_a())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.mem_io_manager'>dagster.mem_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/storage/mem_io_manager.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.mem_io_manager" class="hash-link"></a></Link></dt>
<dd>
Built-in IO manager that stores and retrieves values in memory.
</dd>
</dl>
</div></div>
---
---
title: 'jobs'
sidebar_position: 1000
title_meta: 'jobs API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'jobs Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="jobs">
# Jobs
A `Job` binds a `Graph` and the resources it needs to be executable.
Jobs are created by calling `GraphDefinition.to_job()` on a graph instance, or using the `job` decorator.
<dl>
<dt><Link class="anchor" id='dagster.job'>@dagster.job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/job_decorator.py#L153' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.job" class="hash-link"></a></Link></dt>
<dd>
Creates a job with the specified parameters from the decorated graph/op invocation function.
Using this decorator allows you to build an executable job by writing a function that invokes
ops (or graphs).
Parameters:
- <strong>(</strong><strong>Callable</strong><strong>[</strong><strong>...</strong> (<em>compose_fn</em>) – The decorated function. The body should contain op or graph invocations. Unlike op functions, does not accept a context argument.
- <strong>Any</strong><strong>]</strong> – The decorated function. The body should contain op or graph invocations. Unlike op functions, does not accept a context argument.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name for the Job. Defaults to the name of the this graph.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – Resources that are required by this graph for execution. If not defined, <cite>io_manager</cite> will default to filesystem.
- <strong>config</strong> –
Describes how the job is parameterized at runtime.
If no value is provided, then the schema for the job’s run config is a standard format based on its ops and resources.
If a dictionary is provided, then it must conform to the standard config schema, and it will be used as the job’s run config for the job whenever the job is executed. The values provided will be viewable and editable in the Dagster UI, so be careful with secrets.
If a [`RunConfig`](config.mdx#dagster.RunConfig) object is provided, then it will be used directly as the run config for the job whenever the job is executed, similar to providing a dictionary.
If a [`ConfigMapping`](config.mdx#dagster.ConfigMapping) object is provided, then the schema for the job’s run config is determined by the config mapping, and the ConfigMapping, which should return configuration in the standard format to configure the job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the job and can be used for searching and filtering in the UI. Values that are not already strings will be serialized as JSON. If <cite>run_tags</cite> is not set, then the content of <cite>tags</cite> will also be automatically appended to the tags of any runs of this job.
- <strong>run_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of key-value tags that will be automatically attached to runs launched by this job. Values that are not already strings will be serialized as JSON. These tag values may be overwritten by tag values provided at invocation time. If <cite>run_tags</cite> is set, then <cite>tags</cite> are not automatically appended to the tags of any runs of this job.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary information that will be attached to the JobDefinition and be viewable in the Dagster UI. Keys must be strings, and values must be python primitive types or one of the provided MetadataValue types
- <strong>logger_defs</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – A dictionary of string logger identifiers to their implementations.
- <strong>executor_def</strong> (<em>Optional</em><em>[</em>[*ExecutorDefinition*](internals.mdx#dagster.ExecutorDefinition)<em>]</em>) – How this Job will be executed. Defaults to [`multiprocess_executor`](execution.mdx#dagster.multiprocess_executor) .
- <strong>op_retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](ops.mdx#dagster.RetryPolicy)<em>]</em>) – The default retry policy for all ops in this job. Only used if retry policy is not defined on the op definition or op invocation.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines a discrete set of partition keys that can parameterize the job. If this argument is supplied, the config argument can’t also be supplied.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of a job.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A list of strings representing owners of the job. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
Examples:
```python
@op
def return_one():
return 1
@op
def add_one(in1):
return in1 + 1
@job
def job1():
add_one(return_one())
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition'>`class` dagster.JobDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L726' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a Dagster job.
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.execute_in_process'>execute_in_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L732' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.execute_in_process" class="hash-link"></a></Link></dt>
<dd>
Execute the Job in-process, gathering results in-memory.
The <cite>executor_def</cite> on the Job will be ignored, and replaced with the in-process executor.
If using the default <cite>io_manager</cite>, it will switch from filesystem to in-memory.
Parameters:
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The configuration for the run
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The instance to execute against, an ephemeral one will be used if none provided.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The string partition key that specifies the run config to execute. Can only be used to select run config for jobs with partitioned config.
- <strong>raise_on_error</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to raise exceptions when they occur. Defaults to `True`.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of op selection queries (including single op names) to execute. For example: * `['some_op']`: selects `some_op` itself. * `['*some_op']`: select `some_op` and all its ancestors (upstream dependencies). * `['*some_op+++']`: select `some_op`, all its ancestors, and its descendants (downstream dependencies) within 3 levels down. * `['*some_op', 'other_op_a', 'other_op_b+']`: select `some_op` and all its ancestors, `other_op_a` itself, and `other_op_b` and its direct child ops.
- <strong>input_values</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary that maps python objects to the top-level inputs of the job. Input values provided here will override input values that have been provided to the job directly.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources needed if any are required. Can provide resource instances directly, or resource definitions.
Returns: [`ExecuteInProcessResult`](execution.mdx#dagster.ExecuteInProcessResult)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.run_request_for_partition'>run_request_for_partition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1052' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.run_request_for_partition" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.0.
Directly instantiate `RunRequest(partition_key=...)` instead..
:::
Creates a RunRequest object for a run that processes the given partition.
Parameters:
- <strong>partition_key</strong> – The key of the partition to request a run for.
- <strong>run_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string key to identify this launched run. For sensors, ensures that only one run is created per run key across all sensor evaluations. For schedules, ensures that one run is created per tick, across failure recoveries. Passing in a <cite>None</cite> value means that a run will always be launched per evaluation.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A dictionary of tags (string key-value pairs) to attach to the launched run.
- <strong>(</strong><strong>Optional</strong><strong>[</strong><strong>Mapping</strong><strong>[</strong><strong>str</strong> (<em>run_config</em>) – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>Any</strong><strong>]</strong><strong>]</strong> – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – Used to determine which time-partitions exist. Defaults to now.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Required when the partitions definition is a DynamicPartitionsDefinition with a name defined. Users can pass the DagsterInstance fetched via <cite>context.instance</cite> to this argument.
Returns: an object that requests a run to process the given partition.Return type: [RunRequest](schedules-sensors.mdx#dagster.RunRequest)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.with_hooks'>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1207' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.with_hooks" class="hash-link"></a></Link></dt>
<dd>
Apply a set of hooks to all op instances within the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.with_top_level_resources'>with_top_level_resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L1199' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.with_top_level_resources" class="hash-link"></a></Link></dt>
<dd>
Apply a set of resources to all op instances within the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.config_mapping'>`property` config_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.config_mapping" class="hash-link"></a></Link></dt>
<dd>
The config mapping for the job, if it has one.
A config mapping defines a way to map a top-level config schema to run config for the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.executor_def'>`property` executor_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L402' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.executor_def" class="hash-link"></a></Link></dt>
<dd>
Returns the default [`ExecutorDefinition`](internals.mdx#dagster.ExecutorDefinition) for the job.
If the user has not specified an executor definition, then this will default to the
[`multi_or_in_process_executor()`](execution.mdx#dagster.multi_or_in_process_executor). If a default is specified on the
[`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to, then that will be used instead.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.has_specified_executor'>`property` has_specified_executor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L413' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.has_specified_executor" class="hash-link"></a></Link></dt>
<dd>
Returns True if this job has explicitly specified an executor, and False if the executor
was inherited through defaults or the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.has_specified_loggers'>`property` has_specified_loggers <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L463' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.has_specified_loggers" class="hash-link"></a></Link></dt>
<dd>
Returns true if the job explicitly set loggers, and False if loggers were inherited
through defaults or the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.loggers'>`property` loggers <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.loggers" class="hash-link"></a></Link></dt>
<dd>
Returns the set of LoggerDefinition objects specified on the job.
If the user has not specified a mapping of [`LoggerDefinition`](loggers.mdx#dagster.LoggerDefinition) objects, then this
will default to the `colored_console_logger()` under the key <cite>console</cite>. If a default
is specified on the [`Definitions`](definitions.mdx#dagster.Definitions) object the job was provided to, then that will
be used instead.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.partitioned_config'>`property` partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L431' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.partitioned_config" class="hash-link"></a></Link></dt>
<dd>
The partitioned config for the job, if it has one.
A partitioned config defines a way to map partition keys to run config for the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.partitions_def'>`property` partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L485' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.partitions_def" class="hash-link"></a></Link></dt>
<dd>
Returns the [`PartitionsDefinition`](partitions.mdx#dagster.PartitionsDefinition) for the job, if it has one.
A partitions definition defines the set of partition keys the job operates on.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobDefinition.resource_defs'>`property` resource_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/job_definition.py#L421' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobDefinition.resource_defs" class="hash-link"></a></Link></dt>
<dd>
Returns the set of ResourceDefinition objects specified on the job.
This may not be the complete set of resources required by the job, since those can also be
provided on the [`Definitions`](definitions.mdx#dagster.Definitions) object the job may be provided to.
</dd>
</dl>
</dd>
</dl>
<div class="section" id="reconstructable-jobs">
## Reconstructable jobs
<dl>
<dt>`class` dagster.reconstructable <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/reconstruct.py#L328' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Create a `ReconstructableJob` from a
function that returns a [`JobDefinition`](#dagster.JobDefinition)/[`JobDefinition`](#dagster.JobDefinition),
or a function decorated with [`@job`](#dagster.job).
When your job must cross process boundaries, e.g., for execution on multiple nodes or
in different systems (like `dagstermill`), Dagster must know how to reconstruct the job
on the other side of the process boundary.
Passing a job created with `~dagster.GraphDefinition.to_job` to `reconstructable()`,
requires you to wrap that job’s definition in a module-scoped function, and pass that function
instead:
```python
from dagster import graph, reconstructable
@graph
def my_graph():
...
def define_my_job():
return my_graph.to_job()
reconstructable(define_my_job)
```
This function implements a very conservative strategy for reconstruction, so that its behavior
is easy to predict, but as a consequence it is not able to reconstruct certain kinds of jobs
or jobs, such as those defined by lambdas, in nested scopes (e.g., dynamically within a method
call), or in interactive environments such as the Python REPL or Jupyter notebooks.
If you need to reconstruct objects constructed in these ways, you should use
`build_reconstructable_job()` instead, which allows you to
specify your own reconstruction strategy.
Examples:
```python
from dagster import job, reconstructable
@job
def foo_job():
...
reconstructable_foo_job = reconstructable(foo_job)
@graph
def foo():
...
def make_bar_job():
return foo.to_job()
reconstructable_bar_job = reconstructable(make_bar_job)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_reconstructable_job'>dagster.build_reconstructable_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/reconstruct.py#L444' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_reconstructable_job" class="hash-link"></a></Link></dt>
<dd>
Create a `dagster._core.definitions.reconstructable.ReconstructableJob`.
When your job must cross process boundaries, e.g., for execution on multiple nodes or in
different systems (like `dagstermill`), Dagster must know how to reconstruct the job
on the other side of the process boundary.
This function allows you to use the strategy of your choice for reconstructing jobs, so
that you can reconstruct certain kinds of jobs that are not supported by
[`reconstructable()`](execution.mdx#dagster.reconstructable), such as those defined by lambdas, in nested scopes (e.g.,
dynamically within a method call), or in interactive environments such as the Python REPL or
Jupyter notebooks.
If you need to reconstruct jobs constructed in these ways, use this function instead of
[`reconstructable()`](execution.mdx#dagster.reconstructable).
Parameters:
- <strong>reconstructor_module_name</strong> (<em>str</em>) – The name of the module containing the function to use to reconstruct the job.
- <strong>reconstructor_function_name</strong> (<em>str</em>) – The name of the function to use to reconstruct the job.
- <strong>reconstructable_args</strong> (<em>Tuple</em>) – Args to the function to use to reconstruct the job. Values of the tuple must be JSON serializable.
- <strong>reconstructable_kwargs</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – Kwargs to the function to use to reconstruct the job. Values of the dict must be JSON serializable.
Examples:
```python
# module: mymodule
from dagster import JobDefinition, job, build_reconstructable_job
class JobFactory:
def make_job(self, *args, **kwargs):
@job
def _job():
...
return _job
def reconstruct_job(*args):
factory = JobFactory()
return factory.make_job(*args)
factory = JobFactory()
foo_job_args = (..., ...)
foo_job_kwargs = {...}
foo_job = factory.make_job(*foo_job_args, **foo_job_kwargs)
reconstructable_foo_job = build_reconstructable_job(
'mymodule',
'reconstruct_job',
foo_job_args,
foo_job_kwargs,
)
```
</dd>
</dl>
</div></div>
---
---
title: 'loggers'
sidebar_position: 1000
title_meta: 'loggers API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'loggers Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="loggers">
# Loggers
<div class="section" id="built-in-loggers">
## Built-in loggers
<dl>
<dt><Link class="anchor" id='dagster._loggers.colored_console_logger'>dagster._loggers.colored_console_logger <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._loggers.colored_console_logger" class="hash-link"></a></Link></dt>
<dd>
Core class for defining loggers.
Loggers are job-scoped logging handlers, which will be automatically invoked whenever
dagster messages are logged from within a job.
Parameters:
- <strong>logger_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitLoggerContext*](#dagster.InitLoggerContext)<em>]</em><em>, </em><em>logging.Logger</em><em>]</em>) – User-provided function to instantiate the logger. This logger will be automatically invoked whenever the methods on `context.log` are called from within job compute logic.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this logger.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._loggers.json_console_logger'>dagster._loggers.json_console_logger <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._loggers.json_console_logger" class="hash-link"></a></Link></dt>
<dd>
Core class for defining loggers.
Loggers are job-scoped logging handlers, which will be automatically invoked whenever
dagster messages are logged from within a job.
Parameters:
- <strong>logger_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitLoggerContext*](#dagster.InitLoggerContext)<em>]</em><em>, </em><em>logging.Logger</em><em>]</em>) – User-provided function to instantiate the logger. This logger will be automatically invoked whenever the methods on `context.log` are called from within job compute logic.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this logger.
</dd>
</dl>
</div>
<div class="section" id="logging-from-an-op">
## Logging from an @op
<dl>
<dt><Link class="anchor" id='dagster.DagsterLogManager'>`class` dagster.DagsterLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/log_manager.py#L282' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterLogManager" class="hash-link"></a></Link></dt>
<dd>
Centralized dispatch for logging from user code.
Handles the construction of uniform structured log messages and passes them through to the
underlying loggers/handlers.
An instance of the log manager is made available to ops as `context.log`. Users should not
initialize instances of the log manager directly. To configure custom loggers, set the
`logger_defs` argument in an <cite>@job</cite> decorator or when calling the <cite>to_job()</cite> method on a
[`GraphDefinition`](graphs.mdx#dagster.GraphDefinition).
The log manager inherits standard convenience methods like those exposed by the Python standard
library `python:logging` module (i.e., within the body of an op,
`context.log.\{debug, info, warning, warn, error, critical, fatal}`).
The underlying integer API can also be called directly using, e.g.
`context.log.log(5, msg)`, and the log manager will delegate to the `log` method
defined on each of the loggers it manages.
User-defined custom log levels are not supported, and calls to, e.g.,
`context.log.trace` or `context.log.notice` will result in hard exceptions <strong>at runtime</strong>.
</dd>
</dl>
</div>
<div class="section" id="defining-custom-loggers">
## Defining custom loggers
<dl>
<dt><Link class="anchor" id='dagster.logger'>@dagster.logger <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L133' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.logger" class="hash-link"></a></Link></dt>
<dd>
Define a logger.
The decorated function should accept an [`InitLoggerContext`](#dagster.InitLoggerContext) and return an instance of
`python:logging.Logger`. This function will become the `logger_fn` of an underlying
[`LoggerDefinition`](#dagster.LoggerDefinition).
Parameters:
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the logger.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LoggerDefinition'>`class` dagster.LoggerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LoggerDefinition" class="hash-link"></a></Link></dt>
<dd>
Core class for defining loggers.
Loggers are job-scoped logging handlers, which will be automatically invoked whenever
dagster messages are logged from within a job.
Parameters:
- <strong>logger_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitLoggerContext*](#dagster.InitLoggerContext)<em>]</em><em>, </em><em>logging.Logger</em><em>]</em>) – User-provided function to instantiate the logger. This logger will be automatically invoked whenever the methods on `context.log` are called from within job compute logic.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this logger.
<dl>
<dt><Link class="anchor" id='dagster.LoggerDefinition.config_schema'>`property` config_schema <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LoggerDefinition.config_schema" class="hash-link"></a></Link></dt>
<dd>
The schema for the logger’s config. Configuration data available in <cite>init_context.logger_config</cite>.
Type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LoggerDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L103' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LoggerDefinition.description" class="hash-link"></a></Link></dt>
<dd>
A human-readable description of the logger.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LoggerDefinition.logger_fn'>`property` logger_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LoggerDefinition.logger_fn" class="hash-link"></a></Link></dt>
<dd>
The function that will be invoked to
instantiate the logger.
Type: Callable[[[InitLoggerContext](#dagster.InitLoggerContext)], logging.Logger]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitLoggerContext'>`class` dagster.InitLoggerContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/logger.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitLoggerContext" class="hash-link"></a></Link></dt>
<dd>
The context object available as the argument to the initialization function of a [`dagster.LoggerDefinition`](#dagster.LoggerDefinition).
Users should not instantiate this object directly. To construct an
<cite>InitLoggerContext</cite> for testing purposes, use `dagster.
build_init_logger_context()`.
Example:
```python
from dagster import logger, InitLoggerContext
@logger
def hello_world(init_context: InitLoggerContext):
...
```
<dl>
<dt><Link class="anchor" id='dagster.InitLoggerContext.logger_config'>`property` logger_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/logger.py#L42' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitLoggerContext.logger_config" class="hash-link"></a></Link></dt>
<dd>
The configuration data provided by the run config. The
schema for this data is defined by `config_schema` on the [`LoggerDefinition`](#dagster.LoggerDefinition).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitLoggerContext.logger_def'>`property` logger_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/logger.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitLoggerContext.logger_def" class="hash-link"></a></Link></dt>
<dd>
The logger definition for the logger being constructed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitLoggerContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/logger.py#L61' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitLoggerContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The ID for this run of the job.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_init_logger_context'>dagster.build_init_logger_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L164' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_init_logger_context" class="hash-link"></a></Link></dt>
<dd>
Builds logger initialization context from provided parameters.
This function can be used to provide the context argument to the invocation of a logger
definition.
Note that you may only specify one of pipeline_def and job_def.
Parameters:
- <strong>logger_config</strong> (<em>Any</em>) – The config to provide during initialization of logger.
- <strong>job_def</strong> (<em>Optional</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>]</em>) – The job definition that the logger will be used with.
Examples:
```python
context = build_init_logger_context()
logger_to_init(context)
```
</dd>
</dl>
</div></div>
---
---
title: 'metadata'
sidebar_position: 1000
title_meta: 'metadata API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'metadata Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="metadata">
# Metadata
Dagster uses metadata to communicate arbitrary user-specified metadata about structured
events.
Refer to the [Metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) documentation for more information.
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue'>`class` dagster.MetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L41' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue" class="hash-link"></a></Link></dt>
<dd>
Utility class to wrap metadata values passed into Dagster events so that they can be
displayed in the Dagster UI and other tooling.
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"my_text_label": "hello",
"dashboard_url": MetadataValue.url("http://mycoolsite.com/my_dashboard"),
"num_rows": 0,
},
)
```
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.asset'>`static` asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L344' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.asset" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value referencing a Dagster asset, by key.
For example:
```python
@op
def validate_table(context, df):
yield AssetMaterialization(
asset_key=AssetKey("my_table"),
metadata={
"Related asset": MetadataValue.asset(AssetKey('my_other_table')),
},
)
```
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – The asset key referencing the asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.bool'>`static` bool <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L286' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.bool" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a bool as
`BoolMetadataValuye`. Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"num rows > 1000": MetadataValue.bool(len(df) > 1000),
},
)
```
Parameters: <strong>value</strong> (<em>bool</em>) – The bool value for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.column_lineage'>`static` column_lineage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L474' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.column_lineage" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a column lineage as
[`TableColumnLineageMetadataValue`](#dagster.TableColumnLineageMetadataValue). Can be used as the value type
for the <cite>metadata</cite> parameter for supported events.
Parameters: <strong>lineage</strong> ([*TableColumnLineage*](#dagster.TableColumnLineage)) – The column lineage for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.dagster_run'>`static` dagster_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L334' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.dagster_run" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a reference to a Dagster run.
Parameters: <strong>run_id</strong> (<em>str</em>) – The ID of the run.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.float'>`static` float <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L238' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.float" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a float as
[`FloatMetadataValue`](#dagster.FloatMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"size (bytes)": MetadataValue.float(calculate_bytes(df)),
}
)
```
Parameters: <strong>value</strong> (<em>float</em>) – The float value for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.int'>`static` int <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L262' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.int" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping an int as
[`IntMetadataValue`](#dagster.IntMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"number of rows": MetadataValue.int(len(df)),
},
)
```
Parameters: <strong>value</strong> (<em>int</em>) – The int value for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.job'>`static` job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L370' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.job" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value referencing a Dagster job, by name.
For example:
```python
from dagster import AssetMaterialization, MetadataValue, op
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"Producing job": MetadataValue.job('my_other_job', 'my_location'),
},
)
```
Parameters:
- <strong>job_name</strong> (<em>str</em>) – The name of the job.
- <strong>location_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The code location name for the job.
- <strong>repository_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The repository name of the job, if different from the default.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.json'>`static` json <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L161' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.json" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a json-serializable list or dict
as [`JsonMetadataValue`](#dagster.JsonMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context):
yield ExpectationResult(
success=not missing_things,
label="is_present",
metadata={
"about my dataset": MetadataValue.json({"missing_columns": missing_things})
},
)
```
Parameters: <strong>data</strong> (<em>Union</em><em>[</em><em>Sequence</em><em>[</em><em>Any</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The JSON data for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.md'>`static` md <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L186' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.md" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping markdown data as
[`MarkdownMetadataValue`](#dagster.MarkdownMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Parameters: <strong>data</strong> (<em>str</em>) – The markdown for a metadata entry.
Example:
```python
@op
def emit_metadata(context, md_str):
yield AssetMaterialization(
asset_key="info",
metadata={
'Details': MetadataValue.md(md_str)
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.notebook'>`static` notebook <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L138' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.notebook" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a notebook path as
[`NotebookMetadataValue`](#dagster.NotebookMetadataValue).
Example:
```python
@op
def emit_metadata(context):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"notebook_path": MetadataValue.notebook("path/to/notebook.ipynb"),
}
)
```
Parameters: <strong>path</strong> (<em>str</em>) – The path to a notebook for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.null'>`static` null <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L488' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.null" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value representing null. Can be used as the value type
for the <cite>metadata</cite> parameter for supported events.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.path'>`static` path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L115' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.path" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a path as
[`PathMetadataValue`](#dagster.PathMetadataValue).
Example:
```python
@op
def emit_metadata(context):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"filepath": MetadataValue.path("path/to/file"),
}
)
```
Parameters: <strong>path</strong> (<em>str</em>) – The path for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.pool'>`static` pool <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L508' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.pool" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a reference to a concurrency pool.
Parameters: <strong>pool</strong> (<em>str</em>) – The identifier for the pool.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.python_artifact'>`static` python_artifact <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L211' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.python_artifact" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a python artifact as
[`PythonArtifactMetadataValue`](#dagster.PythonArtifactMetadataValue). Can be used as the value type for the
<cite>metadata</cite> parameter for supported events.
Parameters: <strong>python_artifact</strong> (<em>Callable</em>) – The python class or function for a metadata entry.
Example:
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"class": MetadataValue.python_artifact(MyClass),
"function": MetadataValue.python_artifact(my_function),
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.table'>`static` table <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L407' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.table" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping arbitrary tabular data as
[`TableMetadataValue`](#dagster.TableMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context):
yield ExpectationResult(
success=not has_errors,
label="is_valid",
metadata={
"errors": MetadataValue.table(
records=[
TableRecord(data={"code": "invalid-data-type", "row": 2, "col": "name"})
],
schema=TableSchema(
columns=[
TableColumn(name="code", type="string"),
TableColumn(name="row", type="int"),
TableColumn(name="col", type="string"),
]
)
),
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.table_schema'>`static` table_schema <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L442' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.table_schema" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a table schema as
[`TableSchemaMetadataValue`](#dagster.TableSchemaMetadataValue). Can be used as the value type
for the <cite>metadata</cite> parameter for supported events.
Example:
```python
schema = TableSchema(
columns = [
TableColumn(name="id", type="int"),
TableColumn(name="status", type="bool"),
]
)
DagsterType(
type_check_fn=some_validation_fn,
name='MyTable',
metadata={
'my_table_schema': MetadataValue.table_schema(schema),
}
)
```
Parameters: <strong>schema</strong> ([*TableSchema*](#dagster.TableSchema)) – The table schema for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.text'>`static` text <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L67' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.text" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping text as
[`TextMetadataValue`](#dagster.TextMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context, df):
yield AssetMaterialization(
asset_key="my_dataset",
metadata={
"my_text_label": MetadataValue.text("hello")
},
)
```
Parameters: <strong>text</strong> (<em>str</em>) – The text string for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.timestamp'>`static` timestamp <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L310' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.timestamp" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a UNIX timestamp as a
[`TimestampMetadataValue`](#dagster.TimestampMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Parameters: <strong>value</strong> (<em>Union</em><em>[</em><em>float</em><em>, </em><em>datetime</em><em>]</em>) – The unix timestamp value for a metadata entry. If a
datetime is provided, the timestamp will be extracted. datetimes without timezones
are not accepted, because their timestamps can be ambiguous.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.url'>`static` url <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L91' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.url" class="hash-link"></a></Link></dt>
<dd>
Static constructor for a metadata value wrapping a URL as
[`UrlMetadataValue`](#dagster.UrlMetadataValue). Can be used as the value type for the <cite>metadata</cite>
parameter for supported events.
Example:
```python
@op
def emit_metadata(context):
yield AssetMaterialization(
asset_key="my_dashboard",
metadata={
"dashboard_url": MetadataValue.url("http://mycoolsite.com/my_dashboard"),
}
)
```
Parameters: <strong>url</strong> (<em>str</em>) – The URL for a metadata entry.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataValue.value'>`abstract` `property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L60' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped value.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MetadataEntry'>`class` dagster.MetadataEntry <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/__init__.py#L227' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MetadataEntry" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Please use a dict with `MetadataValue` values instead..
:::
A structure for describing metadata for Dagster events.
Note: This class is no longer usable in any Dagster API, and will be completely removed in 2.0.
Lists of objects of this type can be passed as arguments to Dagster events and will be displayed
in the Dagster UI and other tooling.
Should be yielded from within an IO manager to append metadata for a given input/output event.
For other event types, passing a dict with <cite>MetadataValue</cite> values to the <cite>metadata</cite> argument
is preferred.
Parameters:
- <strong>label</strong> (<em>str</em>) – Short display label for this metadata entry.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this metadata entry.
- <strong>value</strong> ([*MetadataValue*](#dagster.MetadataValue)) – Typed metadata entry data. The different types allow for customized display in tools like the Dagster UI.
</dd>
</dl>
<div class="section" id="metadata-types">
## Metadata types
All metadata types inherit from <cite>MetadataValue</cite>. The following types are defined:
<dl>
<dt><Link class="anchor" id='dagster.DagsterAssetMetadataValue'>`class` dagster.DagsterAssetMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L824' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterAssetMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Representation of a dagster asset.
Parameters: <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – The dagster asset key
<dl>
<dt><Link class="anchor" id='dagster.DagsterAssetMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L836' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterAssetMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped [`AssetKey`](assets.mdx#dagster.AssetKey).
Type: [AssetKey](assets.mdx#dagster.AssetKey)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunMetadataValue'>`class` dagster.DagsterRunMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L783' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRunMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Representation of a dagster run.
Parameters: <strong>run_id</strong> (<em>str</em>) – The run id
<dl>
<dt><Link class="anchor" id='dagster.DagsterRunMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L795' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterRunMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped run id.
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FloatMetadataValue'>`class` dagster.FloatMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L732' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FloatMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for float metadata entry data.
Parameters: <strong>value</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – The float value.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IntMetadataValue'>`class` dagster.IntMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L745' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IntMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for int metadata entry data.
Parameters: <strong>value</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The int value.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JsonMetadataValue'>`class` dagster.JsonMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L654' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JsonMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for JSON metadata entry data.
Parameters: <strong>data</strong> (<em>Union</em><em>[</em><em>Sequence</em><em>[</em><em>Any</em><em>]</em><em>, </em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The JSON data.
<dl>
<dt><Link class="anchor" id='dagster.JsonMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L681' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JsonMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped JSON data.
Type: Optional[Union[Sequence[Any], Dict[str, Any]]]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MarkdownMetadataValue'>`class` dagster.MarkdownMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L688' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MarkdownMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for markdown metadata entry data.
Parameters: <strong>md_str</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The markdown as a string.
<dl>
<dt><Link class="anchor" id='dagster.MarkdownMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L700' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MarkdownMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped markdown as a string.
Type: Optional[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PathMetadataValue'>`class` dagster.PathMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L569' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PathMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for path metadata entry data.
Parameters: <strong>path</strong> (<em>str</em>) – The path as a string or conforming to os.PathLike.
<dl>
<dt><Link class="anchor" id='dagster.PathMetadataValue.path'>`property` path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L594' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PathMetadataValue.path" class="hash-link"></a></Link></dt>
<dd>
The wrapped path.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PathMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L588' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PathMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped path.
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.NotebookMetadataValue'>`class` dagster.NotebookMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L601' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.NotebookMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for notebook metadata entry data.
Parameters: <strong>path</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The path to the notebook as a string or conforming to os.PathLike.
<dl>
<dt><Link class="anchor" id='dagster.NotebookMetadataValue.path'>`property` path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L626' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.NotebookMetadataValue.path" class="hash-link"></a></Link></dt>
<dd>
The wrapped path to the notebook as a string.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.NotebookMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L620' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.NotebookMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped path to the notebook as a string.
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PythonArtifactMetadataValue'>`class` dagster.PythonArtifactMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L708' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PythonArtifactMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for python artifact metadata entry data.
Parameters:
- <strong>module</strong> (<em>str</em>) – The module where the python artifact can be found
- <strong>name</strong> (<em>str</em>) – The name of the python artifact
<dl>
<dt><Link class="anchor" id='dagster.PythonArtifactMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L725' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PythonArtifactMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
Identity function.
Type: [PythonArtifactMetadataValue](#dagster.PythonArtifactMetadataValue)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumnLineageMetadataValue'>`class` dagster.TableColumnLineageMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L939' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnLineageMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Representation of the lineage of column inputs to column outputs of arbitrary tabular data.
Parameters: <strong>column_lineage</strong> ([*TableColumnLineage*](#dagster.TableColumnLineage)) – The lineage of column inputs to column outputs
for the table.
<dl>
<dt><Link class="anchor" id='dagster.TableColumnLineageMetadataValue.column_lineage'>`property` column_lineage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L967' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnLineageMetadataValue.column_lineage" class="hash-link"></a></Link></dt>
<dd>
The wrapped column lineage.
Type: [TableColumnLineage](#dagster.TableColumnLineage)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumnLineageMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L961' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnLineageMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped `TableSpec`.
Type: TableSpec
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableMetadataValue'>`class` dagster.TableMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L844' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for table metadata entry data.
Parameters:
- <strong>records</strong> ([*TableRecord*](#dagster.TableRecord)) – The data as a list of records (i.e. rows).
- <strong>schema</strong> (<em>Optional</em><em>[</em>[*TableSchema*](#dagster.TableSchema)<em>]</em>) – A schema for the table.
Example:
```python
from dagster import TableMetadataValue, TableRecord
TableMetadataValue(
schema=None,
records=[
TableRecord({"column1": 5, "column2": "x"}),
TableRecord({"column1": 7, "column2": "y"}),
]
)
```
<dl>
<dt><Link class="anchor" id='dagster.TableMetadataValue.infer_column_type'>`static` infer_column_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L875' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableMetadataValue.infer_column_type" class="hash-link"></a></Link></dt>
<dd>
str: Infer the [`TableSchema`](#dagster.TableSchema) column type that will be used for a value.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L913' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
Identity function.
Type: [TableMetadataValue](#dagster.TableMetadataValue)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableSchemaMetadataValue'>`class` dagster.TableSchemaMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L920' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableSchemaMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Representation of a schema for arbitrary tabular data.
Parameters: <strong>schema</strong> ([*TableSchema*](#dagster.TableSchema)) – The dictionary containing the schema representation.
<dl>
<dt><Link class="anchor" id='dagster.TableSchemaMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L932' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableSchemaMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped [`TableSchema`](#dagster.TableSchema).
Type: [TableSchema](#dagster.TableSchema)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TextMetadataValue'>`class` dagster.TextMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L531' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TextMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for text metadata entry data.
Parameters: <strong>text</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The text data.
<dl>
<dt><Link class="anchor" id='dagster.TextMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L543' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TextMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped text data.
Type: Optional[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimestampMetadataValue'>`class` dagster.TimestampMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L770' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimestampMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for metadata value that’s a unix timestamp.
Parameters: <strong>value</strong> (<em>float</em>) – Seconds since the unix epoch.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.UrlMetadataValue'>`class` dagster.UrlMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L550' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.UrlMetadataValue" class="hash-link"></a></Link></dt>
<dd>
Container class for URL metadata entry data.
Parameters: <strong>url</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The URL as a string.
<dl>
<dt><Link class="anchor" id='dagster.UrlMetadataValue.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/metadata_value.py#L562' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.UrlMetadataValue.value" class="hash-link"></a></Link></dt>
<dd>
The wrapped URL.
Type: Optional[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.CodeReferencesMetadataValue'>`class` dagster.CodeReferencesMetadataValue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.CodeReferencesMetadataValue" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Metadata value type which represents source locations (locally or otherwise)
of the asset in question. For example, the file path and line number where the
asset is defined.
Parameters: <strong>sources</strong> (<em>List</em><em>[</em><em>Union</em><em>[</em><em>LocalFileCodeReference</em><em>, </em><em>SourceControlCodeReference</em><em>]</em><em>]</em>) – A list of code references for the asset, such as file locations or
references to source control.
</dd>
</dl>
</div>
<div class="section" id="tables">
## Tables
These APIs provide the ability to express column schemas (<cite>TableSchema</cite>), rows/records (<cite>TableRecord</cite>), and column lineage (<cite>TableColumnLineage</cite>) in Dagster as metadata.
<dl>
<dt><Link class="anchor" id='dagster.TableRecord'>`class` dagster.TableRecord <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableRecord" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableSchema'>`class` dagster.TableSchema <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L142' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableSchema" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster.TableSchema.from_name_type_dict'>`static` from_name_type_dict <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L216' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableSchema.from_name_type_dict" class="hash-link"></a></Link></dt>
<dd>
Constructs a TableSchema from a dictionary whose keys are column names and values are the
names of data types of those columns.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableConstraints'>`class` dagster.TableConstraints <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L31' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableConstraints" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumn'>`class` dagster.TableColumn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumn" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumnConstraints'>`class` dagster.TableColumnConstraints <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnConstraints" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumnLineage'>`class` dagster.TableColumnLineage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L255' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnLineage" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TableColumnDep'>`class` dagster.TableColumnDep <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/table.py#L234' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TableColumnDep" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
<div class="section" id="code-references">
## Code references
The following functions are used to attach source code references to your assets.
For more information, see [Linking assets with source code](https://docs.dagster.io/guides/build/assets/metadata-and-tags#source-code).
<dl>
<dt><Link class="anchor" id='dagster.with_source_code_references'>dagster.with_source_code_references <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L415' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.with_source_code_references" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Wrapper function which attaches local code reference metadata to the provided asset definitions.
This points to the filepath and line number where the asset body is defined.
Parameters: <strong>assets_defs</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>, </em><em>CacheableAssetsDefinition</em><em>]</em><em>]</em>) – The asset definitions to which source code metadata should be attached.Returns: The asset definitions with source code metadata attached.Return type: Sequence[[AssetsDefinition](assets.mdx#dagster.AssetsDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.link_code_references_to_git'>dagster.link_code_references_to_git <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L358' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.link_code_references_to_git" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Wrapper function which converts local file path code references to source control URLs
based on the provided source control URL and branch.
Parameters:
- <strong>assets_defs</strong> (<em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>, </em><em>CacheableAssetsDefinition</em><em>]</em><em>]</em>) – The asset definitions to which source control metadata should be attached. Only assets with local file code references (such as those created by <cite>with_source_code_references</cite>) will be converted.
- <strong>git_url</strong> (<em>str</em>) – The base URL for the source control system. For example, “[https://github.com/dagster-io/dagster](https://github.com/dagster-io/dagster)”.
- <strong>git_branch</strong> (<em>str</em>) – The branch in the source control system, such as “master”.
- <strong>platform</strong> (<em>str</em>) – The hosting platform for the source control system, “github” or “gitlab”. If None, it will be inferred based on <cite>git_url</cite>.
- <strong>file_path_mapping</strong> ([*FilePathMapping*](#dagster.FilePathMapping)) – Specifies the mapping between local file paths and their corresponding paths in a source control repository. Simple usage is to provide a <cite>AnchorBasedFilePathMapping</cite> instance, which specifies an anchor file in the repository and the corresponding local file path, which is extrapolated to all other local file paths. Alternatively, a custom function can be provided which takes a local file path and returns the corresponding path in the repository, allowing for more complex mappings.
Example:
```python
Definitions(
assets=link_code_references_to_git(
with_source_code_references([my_dbt_assets]),
git_url="https://github.com/dagster-io/dagster",
git_branch="master",
platform="github",
file_path_mapping=AnchorBasedFilePathMapping(
local_file_anchor=Path(__file__),
file_anchor_path_in_repository="python_modules/my_module/my-module/__init__.py",
),
)
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.FilePathMapping'>`class` dagster.FilePathMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L198' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FilePathMapping" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Base class which defines a file path mapping function. These functions are used to map local file paths
to their corresponding paths in a source control repository.
In many cases where a source control repository is reproduced exactly on a local machine, the included
AnchorBasedFilePathMapping class can be used to specify a direct mapping between the local file paths and the
repository paths. However, in cases where the repository structure differs from the local structure, a custom
mapping function can be provided to handle these cases.
<dl>
<dt><Link class="anchor" id='dagster.FilePathMapping.convert_to_source_control_path'>`abstractmethod` convert_to_source_control_path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L210' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.FilePathMapping.convert_to_source_control_path" class="hash-link"></a></Link></dt>
<dd>
Maps a local file path to the corresponding path in a source control repository.
Parameters: <strong>local_path</strong> (<em>Path</em>) – The local file path to map.Returns: The corresponding path in the hosted source control repository, relative to the repository root.Return type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AnchorBasedFilePathMapping'>`class` dagster.AnchorBasedFilePathMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L223' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AnchorBasedFilePathMapping" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Specifies the mapping between local file paths and their corresponding paths in a source control repository,
using a specific file “anchor” as a reference point. All other paths are calculated relative to this anchor file.
For example, if the chosen anchor file is <cite>/Users/dagster/Documents/python_modules/my_module/my-module/__init__.py</cite>
locally, and <cite>python_modules/my_module/my-module/__init__.py</cite> in a source control repository, in order to map a
different file <cite>/Users/dagster/Documents/python_modules/my_module/my-module/my_asset.py</cite> to the repository path,
the mapping function will position the file in the repository relative to the anchor file’s position in the repository,
resulting in <cite>python_modules/my_module/my-module/my_asset.py</cite>.
Parameters:
- <strong>local_file_anchor</strong> (<em>Path</em>) – The path to a local file that is present in the repository.
- <strong>file_anchor_path_in_repository</strong> (<em>str</em>) – The path to the anchor file in the repository.
Example:
```python
mapping_fn = AnchorBasedFilePathMapping(
local_file_anchor=Path(__file__),
file_anchor_path_in_repository="python_modules/my_module/my-module/__init__.py",
)
```
<dl>
<dt><Link class="anchor" id='dagster.AnchorBasedFilePathMapping.convert_to_source_control_path'>convert_to_source_control_path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/metadata/source_code.py#L252' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AnchorBasedFilePathMapping.convert_to_source_control_path" class="hash-link"></a></Link></dt>
<dd>
Maps a local file path to the corresponding path in a source control repository
based on the anchor file and its corresponding path in the repository.
Parameters: <strong>local_path</strong> (<em>Path</em>) – The local file path to map.Returns: The corresponding path in the hosted source control repository, relative to the repository root.Return type: str
</dd>
</dl>
</dd>
</dl>
</div></div>
---
---
title: 'ops'
sidebar_position: 1000
title_meta: 'ops API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'ops Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="ops">
# Ops
The foundational unit of computation in Dagster.
<div class="section" id="defining-ops">
## Defining ops
<dl>
<dt><Link class="anchor" id='dagster.op'>@dagster.op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/op_decorator.py#L161' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.op" class="hash-link"></a></Link></dt>
<dd>
Create an op with the specified parameters from the decorated function.
Ins and outs will be inferred from the type signature of the decorated function
if not explicitly provided.
The decorated function will be used as the op’s compute function. The signature of the
decorated function is more flexible than that of the `compute_fn` in the core API; it may:
1. Return a value. This value will be wrapped in an [`Output`](#dagster.Output) and yielded by the compute function.
2. Return an [`Output`](#dagster.Output). This output will be yielded by the compute function.
3. Yield [`Output`](#dagster.Output) or other [event objects](#events)`event objects`. Same as default compute behavior.
Note that options 1) and 2) are incompatible with yielding other events – if you would like
to decorate a function that yields events, it must also wrap its eventual output in an
[`Output`](#dagster.Output) and yield it.
@op supports `async def` functions as well, including async generators when yielding multiple
events or outputs. Note that async ops will generally be run on their own unless using a custom
[`Executor`](internals.mdx#dagster.Executor) implementation that supports running them together.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of op. Must be unique within any [`GraphDefinition`](graphs.mdx#dagster.GraphDefinition) using the op.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of this op. If not provided, and the decorated function has docstring, that docstring will be used as the description.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*In*](#dagster.In)<em>]</em><em>]</em>) – Information about the inputs to the op. Information provided here will be combined with what can be inferred from the function signature.
- <strong>out</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*Out*](#dagster.Out)<em>, </em><em>Dict</em><em>[</em><em>str</em><em>, </em>[*Out*](#dagster.Out)<em>]</em><em>]</em><em>]</em>) – Information about the op outputs. Information provided here will be combined with what can be inferred from the return type signature if the function does not use yield.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The schema for the config. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource handles required by this op.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Arbitrary metadata for the op. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the logic encapsulated by the op. If set, this is used as a default version for all outputs.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](#dagster.RetryPolicy)<em>]</em>) – The retry policy for this op.
Examples:
```python
@op
def hello_world():
print('hello')
@op
def echo(msg: str) -> str:
return msg
@op(
ins={'msg': In(str)},
out=Out(str)
)
def echo_2(msg): # same as above
return msg
@op(
out={'word': Out(), 'num': Out()}
)
def multi_out() -> Tuple[str, int]:
return 'cool', 4
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition'>`class` dagster.OpDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L472' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines an op, the functional unit of user-defined computation.
End users should prefer the [`@op`](#dagster.op) decorator. OpDefinition is generally intended to be
used by framework authors or for programatically generated ops.
Parameters:
- <strong>name</strong> (<em>str</em>) – Name of the op. Must be unique within any [`GraphDefinition`](graphs.mdx#dagster.GraphDefinition) or [`JobDefinition`](jobs.mdx#dagster.JobDefinition) that contains the op.
- <strong>input_defs</strong> (<em>List</em><em>[</em><em>InputDefinition</em><em>]</em>) – Inputs of the op.
- <strong>compute_fn</strong> (<em>Callable</em>) –
The core of the op, the function that performs the actual computation. The signature of this function is determined by `input_defs`, and optionally, an injected first argument, `context`, a collection of information provided by the system.
- <strong>output_defs</strong> (<em>List</em><em>[</em><em>OutputDefinition</em><em>]</em>) – Outputs of the op.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The schema for the config. If set, Dagster will check that the config provided for the op matches this schema and will fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the op.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Arbitrary metadata for the op. Frameworks may expect and require certain metadata to be attached to a op. Users should generally not set metadata directly. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resources handles required by this op.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the code encapsulated by the op. If set, this is used as a default code version for all outputs.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](#dagster.RetryPolicy)<em>]</em>) – The retry policy for this op.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the pool that governs this op’s execution.
Examples:
```python
def _add_one(_context, inputs):
yield Output(inputs["num"] + 1)
OpDefinition(
name="add_one",
ins={"num": In(int)},
outs={"result": Out(int)},
compute_fn=_add_one,
)
```
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.alias'>alias <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L291' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.alias" class="hash-link"></a></Link></dt>
<dd>
Creates a copy of this op with the given name.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.tag'>tag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L296' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.tag" class="hash-link"></a></Link></dt>
<dd>
Creates a copy of this op with the given tags.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.with_hooks'>with_hooks <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L301' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.with_hooks" class="hash-link"></a></Link></dt>
<dd>
Creates a copy of this op with the given hook definitions.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.with_retry_policy'>with_retry_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L306' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.with_retry_policy" class="hash-link"></a></Link></dt>
<dd>
Creates a copy of this op with the given retry policy.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.config_schema'>`property` config_schema <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L258' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.config_schema" class="hash-link"></a></Link></dt>
<dd>
The config schema for this op.
Type: IDefinitionConfigSchema
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.ins'>`property` ins <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L242' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.ins" class="hash-link"></a></Link></dt>
<dd>
A mapping from input name to the In object that represents that input.
Type: Mapping[str, [In](#dagster.In)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L236' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.name" class="hash-link"></a></Link></dt>
<dd>
The name of this op.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.outs'>`property` outs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L248' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.outs" class="hash-link"></a></Link></dt>
<dd>
A mapping from output name to the Out object that represents that output.
Type: Mapping[str, [Out](#dagster.Out)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L264' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
A set of keys for resources that must be provided to this OpDefinition.
Type: AbstractSet[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.retry_policy'>`property` retry_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L279' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.retry_policy" class="hash-link"></a></Link></dt>
<dd>
The RetryPolicy for this op.
Type: Optional[[RetryPolicy](#dagster.RetryPolicy)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.tags'>`property` tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/op_definition.py#L285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.tags" class="hash-link"></a></Link></dt>
<dd>
The tags for this op.
Type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.OpDefinition.version'>`property` version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L270' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.OpDefinition.version" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `code_version` instead..
:::
Version of the code encapsulated by the op. If set, this is used as a
default code version for all outputs.
Type: str
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="ins-outs">
## Ins & outs
<dl>
<dt><Link class="anchor" id='dagster.In'>`class` dagster.In <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/input.py#L399' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.In" class="hash-link"></a></Link></dt>
<dd>
Defines an argument to an op’s compute function.
Inputs may flow from previous op’s outputs, or be stubbed using config. They may optionally
be typed using the Dagster type system.
Parameters:
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Type</em><em>, </em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em><em>]</em><em>]</em>) – The type of this input. Should only be set if the correct type can not be inferred directly from the type signature of the decorated function.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the input.
- <strong>default_value</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The default value to use if no input is provided.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – A dict of metadata for the input.
- <strong>asset_key</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>InputContext -> AssetKey</em><em>]</em><em>]</em>) – An AssetKey (or function that produces an AssetKey from the InputContext) which should be associated with this In. Used for tracking lineage information through Dagster.
- <strong>asset_partitions</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>, </em><em>InputContext -> Set</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – A set of partitions of the given asset_key (or a function that produces this list of partitions from the InputContext) which should be associated with this In.
- <strong>input_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The resource key for the [`InputManager`](io-managers.mdx#dagster.InputManager) used for loading this input when it is not connected to an upstream output.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Out'>`class` dagster.Out <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/output.py#L323' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Out" class="hash-link"></a></Link></dt>
<dd>
Defines an output from an op’s compute function.
Ops can have multiple outputs, in which case outputs cannot be anonymous.
Many ops have only one output, in which case the user can provide a single output definition
that will be given the default name, “result”.
Outs may be typed using the Dagster type system.
Parameters:
- <strong>dagster_type</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Type</em><em>, </em>[*DagsterType*](types.mdx#dagster.DagsterType)<em>]</em><em>]</em><em>]</em>) – The type of this output. Should only be set if the correct type can not be inferred directly from the type signature of the decorated function.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Human-readable description of the output.
- <strong>is_required</strong> (<em>bool</em>) – Whether the presence of this field is required. (default: True)
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The resource key of the output manager used for this output. (default: “io_manager”).
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of the metadata for the output. For example, users can provide a file path if the data object will be stored in a filesystem, or provide information of a database table when it is going to load the data into the table.
- <strong>code_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Version of the code that generates this output. In general, versions should be set only for code that deterministically produces the same output when given the same inputs.
</dd>
</dl>
</div>
<div class="section" id="execution">
## Execution
<dl>
<dt><Link class="anchor" id='dagster.RetryPolicy'>`class` dagster.RetryPolicy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/policy.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RetryPolicy" class="hash-link"></a></Link></dt>
<dd>
A declarative policy for when to request retries when an exception occurs during op execution.
Parameters:
- <strong>max_retries</strong> (<em>int</em>) – The maximum number of retries to attempt. Defaults to 1.
- <strong>delay</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>int</em><em>,</em><em>float</em><em>]</em><em>]</em>) – The time in seconds to wait between the retry being requested and the next attempt being started. This unit of time can be modulated as a function of attempt number with backoff and randomly with jitter.
- <strong>backoff</strong> (<em>Optional</em><em>[</em>[*Backoff*](#dagster.Backoff)<em>]</em>) – A modifier for delay as a function of retry attempt number.
- <strong>jitter</strong> (<em>Optional</em><em>[</em>[*Jitter*](#dagster.Jitter)<em>]</em>) – A randomizing modifier for delay, applied after backoff calculation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Backoff'>`class` dagster.Backoff <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/policy.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Backoff" class="hash-link"></a></Link></dt>
<dd>
A modifier for delay as a function of attempt number.
LINEAR: <cite>attempt_num * delay</cite>
EXPONENTIAL: <cite>((2 ^ attempt_num) - 1) * delay</cite>
<dl>
<dt><Link class="anchor" id='dagster.Backoff.EXPONENTIAL'>EXPONENTIAL `=` 'EXPONENTIAL'<a href="#dagster.Backoff.EXPONENTIAL" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Backoff.LINEAR'>LINEAR `=` 'LINEAR'<a href="#dagster.Backoff.LINEAR" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Jitter'>`class` dagster.Jitter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/policy.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Jitter" class="hash-link"></a></Link></dt>
<dd>
A randomizing modifier for delay, applied after backoff calculation.
FULL: between 0 and the calculated delay based on backoff: <cite>random() * backoff_delay</cite>
PLUS_MINUS: +/- the delay: <cite>backoff_delay + ((2 * (random() * delay)) - delay)</cite>
<dl>
<dt><Link class="anchor" id='dagster.Jitter.FULL'>FULL `=` 'FULL'<a href="#dagster.Jitter.FULL" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Jitter.PLUS_MINUS'>PLUS_MINUS `=` 'PLUS_MINUS'<a href="#dagster.Jitter.PLUS_MINUS" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="events">
<Link id="id1"></Link>
## Events
The objects that can be yielded by the body of ops’ compute functions to communicate with the
Dagster framework.
(Note that [`Failure`](#dagster.Failure) and [`RetryRequested`](#dagster.RetryRequested) are intended to be raised from ops rather than yielded.)
<div class="section" id="event-types">
### Event types
<dl>
<dt><Link class="anchor" id='dagster.Output'>`class` dagster.Output <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L92' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Output" class="hash-link"></a></Link></dt>
<dd>
Event corresponding to one of an op’s outputs.
Op compute functions must explicitly yield events of this type when they have more than
one output, or when they also yield events of other types, or when defining a op using the
[`OpDefinition`](#dagster.OpDefinition) API directly.
Outputs are values produced by ops that will be consumed by downstream ops in a job.
They are type-checked at op boundaries when their corresponding [`Out`](#dagster.Out)
or the downstream [`In`](#dagster.In) is typed.
Parameters:
- <strong>value</strong> (<em>Any</em>) – The value returned by the compute function.
- <strong>output_name</strong> (<em>str</em>) – Name of the corresponding Out. (default: “result”)
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>str</em><em>, </em><em>float</em><em>, </em><em>int</em><em>, </em>[*MetadataValue*](metadata.mdx#dagster.MetadataValue)<em>]</em><em>]</em><em>]</em>) – Arbitrary metadata about the output. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
- <strong>data_version</strong> (<em>Optional</em><em>[</em><em>DataVersion</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A data version to manually set for the asset.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Tags that will be attached to the asset materialization event corresponding to this output, if there is one.
<dl>
<dt><Link class="anchor" id='dagster.Output.data_version'>`property` data_version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L155' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Output.data_version" class="hash-link"></a></Link></dt>
<dd>
A data version that was manually set on the <cite>Output</cite>.
Type: Optional[DataVersion]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Output.output_name'>`property` output_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L149' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Output.output_name" class="hash-link"></a></Link></dt>
<dd>
Name of the corresponding [`Out`](#dagster.Out).
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Output.value'>`property` value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L143' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Output.value" class="hash-link"></a></Link></dt>
<dd>
The value returned by the compute function.
Type: Any
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetMaterialization'>`class` dagster.AssetMaterialization <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L467' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetMaterialization" class="hash-link"></a></Link></dt>
<dd>
Event indicating that an op has materialized an asset.
Op compute functions may yield events of this type whenever they wish to indicate to the
Dagster framework (and the end user) that they have produced a materialized value as a
side effect of computation. Unlike outputs, asset materializations can not be passed to other
ops, and their persistence is controlled by op logic, rather than by the Dagster
framework.
Op authors should use these events to organize metadata about the side effects of their
computations, enabling tooling like the Assets dashboard in the Dagster UI.
Parameters:
- <strong>asset_key</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>, </em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – A key to identify the materialized asset across job runs
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A longer human-readable description of the materialized value.
- <strong>partition</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the partition that was materialized.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A mapping containing tags for the materialization.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the asset. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
<dl>
<dt><Link class="anchor" id='dagster.AssetMaterialization.file'>`static` file <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L561' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetMaterialization.file" class="hash-link"></a></Link></dt>
<dd>
Static constructor for standard materializations corresponding to files on disk.
Parameters:
- <strong>path</strong> (<em>str</em>) – The path to the file.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the materialization.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ExpectationResult'>`class` dagster.ExpectationResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L595' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ExpectationResult" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
If using assets, use AssetCheckResult and @asset_check instead..
:::
Event corresponding to a data quality test.
Op compute functions may yield events of this type whenever they wish to indicate to the
Dagster framework (and the end user) that a data quality test has produced a (positive or
negative) result.
Parameters:
- <strong>success</strong> (<em>bool</em>) – Whether the expectation passed or not.
- <strong>label</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Short display name for expectation. Defaults to “result”.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A longer human-readable description of the expectation.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TypeCheck'>`class` dagster.TypeCheck <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L651' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TypeCheck" class="hash-link"></a></Link></dt>
<dd>
Event corresponding to a successful typecheck.
Events of this type should be returned by user-defined type checks when they need to encapsulate
additional metadata about a type check’s success or failure. (i.e., when using
`as_dagster_type()`, `@usable_as_dagster_type`, or the underlying
[`PythonObjectDagsterType()`](types.mdx#dagster.PythonObjectDagsterType) API.)
Op compute functions should generally avoid yielding events of this type to avoid confusion.
Parameters:
- <strong>success</strong> (<em>bool</em>) – `True` if the type check succeeded, `False` otherwise.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the type check.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.Failure'>`class` dagster.Failure <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L701' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.Failure" class="hash-link"></a></Link></dt>
<dd>
Event indicating op failure.
Raise events of this type from within op compute functions or custom type checks in order to
indicate an unrecoverable failure in user code to the Dagster machinery and return
structured metadata about the failure.
Parameters:
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the failure.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
- <strong>allow_retries</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether this Failure should respect the retry policy or bypass it and immediately fail. Defaults to True, respecting the retry policy and allowing retries.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RetryRequested'>`class` dagster.RetryRequested <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/events.py#L734' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RetryRequested" class="hash-link"></a></Link></dt>
<dd>
An exception to raise from an op to indicate that it should be retried.
Parameters:
- <strong>max_retries</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The max number of retries this step should attempt before failing
- <strong>seconds_to_wait</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>float</em><em>,</em><em>int</em><em>]</em><em>]</em>) – Seconds to wait before restarting the step after putting the step in to the up_for_retry state
Example:
```python
@op
def flakes():
try:
flakey_operation()
except Exception as e:
raise RetryRequested(max_retries=3) from e
```
</dd>
</dl>
</div></div></div>
---
---
title: 'partitions'
sidebar_position: 1000
title_meta: 'partitions API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'partitions Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="partitions">
# Partitions
<div class="section" id="partitions-definitions">
## Partitions Definitions
<dl>
<dt><Link class="anchor" id='dagster.PartitionsDefinition'>`class` dagster.PartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/partitions_definition.py#L29' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a set of partitions, which can be attached to a software-defined asset or job.
Abstract class with implementations for different kinds of partitions.
<dl>
<dt><Link class="anchor" id='dagster.PartitionsDefinition.get_partition_keys'>`abstractmethod` get_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/partitions_definition.py#L45' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionsDefinition.get_partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of strings representing the partition keys of the PartitionsDefinition.
Parameters:
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – A datetime object representing the current time, only applicable to time-based partitions definitions.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Required when the partitions definition is a DynamicPartitionsDefinition with a name defined. Users can pass the DagsterInstance fetched via <cite>context.instance</cite> to this argument.
Returns: Sequence[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition'>`class` dagster.HourlyPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window_subclasses.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A set of hourly partitions.
The first partition in the set will start on the start_date at midnight. The last partition
in the set will end before the current time, unless the end_offset argument is set to a
positive number. If minute_offset is provided, the start and end times of each partition
will be minute_offset past the hour.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions. Can provide in either a datetime or string format.
- <strong>end_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>, </em><em>None</em><em>]</em>) – The last date(excluding) in the set of partitions. Default is None. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>. Note that if a non-UTC timezone is used, the date format must include a timezone offset to disambiguate between multiple instances of the same time before and after the Fall DST transition. If the format does not contain this offset, the second instance of the ambiguous time partition key will have the UTC offset automatically appended to it.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The timezone in which each date should exist. Supported strings for timezones are the ones provided by the [IANA time zone database](https://www.iana.org/time-zones) - e.g. “America/Los_Angeles”.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
from datetime import datetime
from dagster import HourlyPartitionsDefinition
# Basic hourly partitions starting at midnight
hourly_partitions = HourlyPartitionsDefinition(start_date=datetime(2022, 3, 12))
# Hourly partitions with 15-minute offset
offset_partitions = HourlyPartitionsDefinition(
start_date=datetime(2022, 3, 12),
minute_offset=15
)
```
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.get_cron_schedule'>get_cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L917' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.get_cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The schedule executes at the cadence specified by the partitioning, but may overwrite
the minute/hour/day offset of the partitioning.
This is useful e.g. if you have partitions that span midnight to midnight but you want to
schedule a job that runs at 2 am.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.day_offset'>`property` day_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L888' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.day_offset" class="hash-link"></a></Link></dt>
<dd>
For a weekly or monthly partitions definition, returns the day to “split” partitions
by. Each partition will start on this day, and end before this day in the following
week/month. Returns 0 if the day_offset parameter is unset in the
WeeklyPartitionsDefinition, MonthlyPartitionsDefinition, or the provided cron schedule.
For weekly partitions, returns a value between 0 (representing Sunday) and 6 (representing
Saturday). Providing a value of 1 means that a partition will exist weekly from Monday to
the following Sunday.
For monthly partitions, returns a value between 0 (the first day of the month) and 31 (the
last possible day of the month).
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.end'>`property` end <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.end" class="hash-link"></a></Link></dt>
<dd>
The end of the last partition’s time window, or None if the partition set is unbounded.
Type: Optional[datetime]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.hour_offset'>`property` hour_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L876' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.hour_offset" class="hash-link"></a></Link></dt>
<dd>
Number of hours past 00:00 to “split” partitions. Defaults to 0.
For example, returns 1 if each partition starts at 01:00.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.minute_offset'>`property` minute_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L864' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.minute_offset" class="hash-link"></a></Link></dt>
<dd>
Number of minutes past the hour to “split” partitions. Defaults to 0.
For example, returns 15 if each partition starts at 15 minutes past the hour.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.schedule_type'>`property` schedule_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.schedule_type" class="hash-link"></a></Link></dt>
<dd>
An enum representing the partition cadence (hourly, daily,
weekly, or monthly).
Type: Optional[ScheduleType]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.HourlyPartitionsDefinition.start'>`property` start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.HourlyPartitionsDefinition.start" class="hash-link"></a></Link></dt>
<dd>
The start of the first partition’s time window.
Type: datetime
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition'>`class` dagster.DailyPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window_subclasses.py#L101' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A set of daily partitions.
The first partition in the set will start at the start_date at midnight. The last partition
in the set will end before the current time, unless the end_offset argument is set to a
positive number. If minute_offset and/or hour_offset are used, the start and end times of
each partition will be hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions. Can provide in either a datetime or string format.
- <strong>end_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>, </em><em>None</em><em>]</em>) – The last date(excluding) in the set of partitions. Default is None. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
DailyPartitionsDefinition(start_date="2022-03-12")
# creates partitions (2022-03-12-00:00, 2022-03-13-00:00), (2022-03-13-00:00, 2022-03-14-00:00), ...
DailyPartitionsDefinition(start_date="2022-03-12", minute_offset=15, hour_offset=16)
# creates partitions (2022-03-12-16:15, 2022-03-13-16:15), (2022-03-13-16:15, 2022-03-14-16:15), ...
```
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.get_cron_schedule'>get_cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L917' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.get_cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The schedule executes at the cadence specified by the partitioning, but may overwrite
the minute/hour/day offset of the partitioning.
This is useful e.g. if you have partitions that span midnight to midnight but you want to
schedule a job that runs at 2 am.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.day_offset'>`property` day_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L888' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.day_offset" class="hash-link"></a></Link></dt>
<dd>
For a weekly or monthly partitions definition, returns the day to “split” partitions
by. Each partition will start on this day, and end before this day in the following
week/month. Returns 0 if the day_offset parameter is unset in the
WeeklyPartitionsDefinition, MonthlyPartitionsDefinition, or the provided cron schedule.
For weekly partitions, returns a value between 0 (representing Sunday) and 6 (representing
Saturday). Providing a value of 1 means that a partition will exist weekly from Monday to
the following Sunday.
For monthly partitions, returns a value between 0 (the first day of the month) and 31 (the
last possible day of the month).
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.end'>`property` end <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.end" class="hash-link"></a></Link></dt>
<dd>
The end of the last partition’s time window, or None if the partition set is unbounded.
Type: Optional[datetime]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.hour_offset'>`property` hour_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L876' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.hour_offset" class="hash-link"></a></Link></dt>
<dd>
Number of hours past 00:00 to “split” partitions. Defaults to 0.
For example, returns 1 if each partition starts at 01:00.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.minute_offset'>`property` minute_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L864' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.minute_offset" class="hash-link"></a></Link></dt>
<dd>
Number of minutes past the hour to “split” partitions. Defaults to 0.
For example, returns 15 if each partition starts at 15 minutes past the hour.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.schedule_type'>`property` schedule_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.schedule_type" class="hash-link"></a></Link></dt>
<dd>
An enum representing the partition cadence (hourly, daily,
weekly, or monthly).
Type: Optional[ScheduleType]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DailyPartitionsDefinition.start'>`property` start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DailyPartitionsDefinition.start" class="hash-link"></a></Link></dt>
<dd>
The start of the first partition’s time window.
Type: datetime
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition'>`class` dagster.WeeklyPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window_subclasses.py#L182' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a set of weekly partitions.
The first partition in the set will start at the start_date. The last partition in the set will
end before the current time, unless the end_offset argument is set to a positive number. If
day_offset is provided, the start and end date of each partition will be day of the week
corresponding to day_offset (0 indexed with Sunday as the start of the week). If
minute_offset and/or hour_offset are used, the start and end times of each partition will be
hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions will Sunday at midnight following start_date. Can provide in either a datetime or string format.
- <strong>end_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>, </em><em>None</em><em>]</em>) – The last date(excluding) in the set of partitions. Default is None. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>day_offset</strong> (<em>int</em>) – Day of the week to “split” the partition. Defaults to 0 (Sunday).
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
WeeklyPartitionsDefinition(start_date="2022-03-12")
# creates partitions (2022-03-13-00:00, 2022-03-20-00:00), (2022-03-20-00:00, 2022-03-27-00:00), ...
WeeklyPartitionsDefinition(start_date="2022-03-12", minute_offset=15, hour_offset=3, day_offset=6)
# creates partitions (2022-03-12-03:15, 2022-03-19-03:15), (2022-03-19-03:15, 2022-03-26-03:15), ...
```
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.get_cron_schedule'>get_cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L917' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.get_cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The schedule executes at the cadence specified by the partitioning, but may overwrite
the minute/hour/day offset of the partitioning.
This is useful e.g. if you have partitions that span midnight to midnight but you want to
schedule a job that runs at 2 am.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.day_offset'>`property` day_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L888' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.day_offset" class="hash-link"></a></Link></dt>
<dd>
For a weekly or monthly partitions definition, returns the day to “split” partitions
by. Each partition will start on this day, and end before this day in the following
week/month. Returns 0 if the day_offset parameter is unset in the
WeeklyPartitionsDefinition, MonthlyPartitionsDefinition, or the provided cron schedule.
For weekly partitions, returns a value between 0 (representing Sunday) and 6 (representing
Saturday). Providing a value of 1 means that a partition will exist weekly from Monday to
the following Sunday.
For monthly partitions, returns a value between 0 (the first day of the month) and 31 (the
last possible day of the month).
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.end'>`property` end <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.end" class="hash-link"></a></Link></dt>
<dd>
The end of the last partition’s time window, or None if the partition set is unbounded.
Type: Optional[datetime]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.hour_offset'>`property` hour_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L876' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.hour_offset" class="hash-link"></a></Link></dt>
<dd>
Number of hours past 00:00 to “split” partitions. Defaults to 0.
For example, returns 1 if each partition starts at 01:00.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.minute_offset'>`property` minute_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L864' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.minute_offset" class="hash-link"></a></Link></dt>
<dd>
Number of minutes past the hour to “split” partitions. Defaults to 0.
For example, returns 15 if each partition starts at 15 minutes past the hour.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.schedule_type'>`property` schedule_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.schedule_type" class="hash-link"></a></Link></dt>
<dd>
An enum representing the partition cadence (hourly, daily,
weekly, or monthly).
Type: Optional[ScheduleType]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.WeeklyPartitionsDefinition.start'>`property` start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.WeeklyPartitionsDefinition.start" class="hash-link"></a></Link></dt>
<dd>
The start of the first partition’s time window.
Type: datetime
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition'>`class` dagster.MonthlyPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window_subclasses.py#L266' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A set of monthly partitions.
The first partition in the set will start at the soonest first of the month after start_date
at midnight. The last partition in the set will end before the current time, unless the
end_offset argument is set to a positive number. If day_offset is provided, the start and
end date of each partition will be day_offset. If minute_offset and/or hour_offset are used,
the start and end times of each partition will be hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions will be midnight the soonest first of the month following start_date. Can provide in either a datetime or string format.
- <strong>end_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>, </em><em>None</em><em>]</em>) – The last date(excluding) in the set of partitions. Default is None. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>day_offset</strong> (<em>int</em>) – Day of the month to “split” the partition. Defaults to 1.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
MonthlyPartitionsDefinition(start_date="2022-03-12")
# creates partitions (2022-04-01-00:00, 2022-05-01-00:00), (2022-05-01-00:00, 2022-06-01-00:00), ...
MonthlyPartitionsDefinition(start_date="2022-03-12", minute_offset=15, hour_offset=3, day_offset=5)
# creates partitions (2022-04-05-03:15, 2022-05-05-03:15), (2022-05-05-03:15, 2022-06-05-03:15), ...
```
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.get_cron_schedule'>get_cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L917' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.get_cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The schedule executes at the cadence specified by the partitioning, but may overwrite
the minute/hour/day offset of the partitioning.
This is useful e.g. if you have partitions that span midnight to midnight but you want to
schedule a job that runs at 2 am.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.day_offset'>`property` day_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L888' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.day_offset" class="hash-link"></a></Link></dt>
<dd>
For a weekly or monthly partitions definition, returns the day to “split” partitions
by. Each partition will start on this day, and end before this day in the following
week/month. Returns 0 if the day_offset parameter is unset in the
WeeklyPartitionsDefinition, MonthlyPartitionsDefinition, or the provided cron schedule.
For weekly partitions, returns a value between 0 (representing Sunday) and 6 (representing
Saturday). Providing a value of 1 means that a partition will exist weekly from Monday to
the following Sunday.
For monthly partitions, returns a value between 0 (the first day of the month) and 31 (the
last possible day of the month).
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.end'>`property` end <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.end" class="hash-link"></a></Link></dt>
<dd>
The end of the last partition’s time window, or None if the partition set is unbounded.
Type: Optional[datetime]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.hour_offset'>`property` hour_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L876' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.hour_offset" class="hash-link"></a></Link></dt>
<dd>
Number of hours past 00:00 to “split” partitions. Defaults to 0.
For example, returns 1 if each partition starts at 01:00.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.minute_offset'>`property` minute_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L864' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.minute_offset" class="hash-link"></a></Link></dt>
<dd>
Number of minutes past the hour to “split” partitions. Defaults to 0.
For example, returns 15 if each partition starts at 15 minutes past the hour.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.schedule_type'>`property` schedule_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.schedule_type" class="hash-link"></a></Link></dt>
<dd>
An enum representing the partition cadence (hourly, daily,
weekly, or monthly).
Type: Optional[ScheduleType]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MonthlyPartitionsDefinition.start'>`property` start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MonthlyPartitionsDefinition.start" class="hash-link"></a></Link></dt>
<dd>
The start of the first partition’s time window.
Type: datetime
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition'>`class` dagster.TimeWindowPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A set of partitions where each partition corresponds to a time window.
The provided cron_schedule determines the bounds of the time windows. E.g. a cron_schedule of
“0 0 \* \* \*” will result in daily partitions that start at midnight and end at midnight of the
following day.
The string partition_key associated with each partition corresponds to the start of the
partition’s time window.
The first partition in the set will start on at the first cron_schedule tick that is equal to
or after the given start datetime. The last partition in the set will end before the current
time, unless the end_offset argument is set to a positive number.
We recommended limiting partition counts for each asset to 100,000 partitions or fewer.
Parameters:
- <strong>cron_schedule</strong> (<em>str</em>) – Determines the bounds of the time windows.
- <strong>start</strong> (<em>datetime</em>) – The first partition in the set will start on at the first cron_schedule tick that is equal to or after this value.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>end</strong> (<em>datetime</em>) – The last partition (excluding) in the set.
- <strong>fmt</strong> (<em>str</em>) – The date format to use for partition_keys. Note that if a non-UTC timezone is used, and the cron schedule repeats every hour or faster, the date format must include a timezone offset to disambiguate between multiple instances of the same time before and after the Fall DST transition. If the format does not contain this offset, the second instance of the ambiguous time partition key will have the UTC offset automatically appended to it.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.get_cron_schedule'>get_cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L917' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.get_cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The schedule executes at the cadence specified by the partitioning, but may overwrite
the minute/hour/day offset of the partitioning.
This is useful e.g. if you have partitions that span midnight to midnight but you want to
schedule a job that runs at 2 am.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.day_offset'>`property` day_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L888' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.day_offset" class="hash-link"></a></Link></dt>
<dd>
For a weekly or monthly partitions definition, returns the day to “split” partitions
by. Each partition will start on this day, and end before this day in the following
week/month. Returns 0 if the day_offset parameter is unset in the
WeeklyPartitionsDefinition, MonthlyPartitionsDefinition, or the provided cron schedule.
For weekly partitions, returns a value between 0 (representing Sunday) and 6 (representing
Saturday). Providing a value of 1 means that a partition will exist weekly from Monday to
the following Sunday.
For monthly partitions, returns a value between 0 (the first day of the month) and 31 (the
last possible day of the month).
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.end'>`property` end <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.end" class="hash-link"></a></Link></dt>
<dd>
The end of the last partition’s time window, or None if the partition set is unbounded.
Type: Optional[datetime]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.hour_offset'>`property` hour_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L876' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.hour_offset" class="hash-link"></a></Link></dt>
<dd>
Number of hours past 00:00 to “split” partitions. Defaults to 0.
For example, returns 1 if each partition starts at 01:00.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.minute_offset'>`property` minute_offset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L864' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.minute_offset" class="hash-link"></a></Link></dt>
<dd>
Number of minutes past the hour to “split” partitions. Defaults to 0.
For example, returns 15 if each partition starts at 15 minutes past the hour.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.schedule_type'>`property` schedule_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.schedule_type" class="hash-link"></a></Link></dt>
<dd>
An enum representing the partition cadence (hourly, daily,
weekly, or monthly).
Type: Optional[ScheduleType]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionsDefinition.start'>`property` start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/time_window.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionsDefinition.start" class="hash-link"></a></Link></dt>
<dd>
The start of the first partition’s time window.
Type: datetime
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindow'>`class` dagster.TimeWindow <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/utils/time_window.py#L38' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindow" class="hash-link"></a></Link></dt>
<dd>
An interval that is closed at the start and open at the end.
Parameters:
- <strong>start</strong> (<em>datetime</em>) – A datetime that marks the start of the window.
- <strong>end</strong> (<em>datetime</em>) – A datetime that marks the end of the window.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StaticPartitionsDefinition'>`class` dagster.StaticPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/static.py#L24' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StaticPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A statically-defined set of partitions.
We recommended limiting partition counts for each asset to 100,000 partitions or fewer.
Example:
```python
from dagster import StaticPartitionsDefinition, asset
oceans_partitions_def = StaticPartitionsDefinition(
["arctic", "atlantic", "indian", "pacific", "southern"]
)
@asset(partitions_def=oceans_partitions_defs)
def ml_model_for_each_ocean():
...
```
<dl>
<dt><Link class="anchor" id='dagster.StaticPartitionsDefinition.get_partition_keys'>get_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/static.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StaticPartitionsDefinition.get_partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of strings representing the partition keys of the PartitionsDefinition.
Parameters:
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – A datetime object representing the current time, only applicable to time-based partitions definitions.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Only applicable to DynamicPartitionsDefinitions.
Returns: Sequence[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiPartitionsDefinition'>`class` dagster.MultiPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/multi.py#L84' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
Takes the cross-product of partitions from two partitions definitions.
For example, with a static partitions definition where the partitions are [“a”, “b”, “c”]
and a daily partitions definition, this partitions definition will have the following
partitions:
2020-01-01|a
2020-01-01|b
2020-01-01|c
2020-01-02|a
2020-01-02|b
…
We recommended limiting partition counts for each asset to 100,000 partitions or fewer.
Parameters:
- <strong>partitions_defs</strong> (<em>Sequence</em><em>[</em><em>PartitionDimensionDefinition</em><em>]</em>) – A mapping of dimension name to partitions definition. The total set of partitions will be the cross-product of the partitions from each PartitionsDefinition.
- <strong>partitions_defs</strong> – A sequence of PartitionDimensionDefinition objects, each of which contains a dimension name and a PartitionsDefinition. The total set of partitions will be the cross-product of the partitions from each PartitionsDefinition. This sequence is ordered by dimension name, to ensure consistent ordering of the partitions.
<dl>
<dt><Link class="anchor" id='dagster.MultiPartitionsDefinition.get_partition_keys'>get_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/multi.py#L232' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiPartitionsDefinition.get_partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of MultiPartitionKeys representing the partition keys of the
PartitionsDefinition.
Parameters:
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – A datetime object representing the current time, only applicable to time-based partition dimensions.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Required when a dimension is a DynamicPartitionsDefinition with a name defined. Users can pass the DagsterInstance fetched via <cite>context.instance</cite> to this argument.
Returns: Sequence[MultiPartitionKey]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiPartitionKey'>`class` dagster.MultiPartitionKey <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/utils/multi.py#L134' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiPartitionKey" class="hash-link"></a></Link></dt>
<dd>
A multi-dimensional partition key stores the partition key for each dimension.
Subclasses the string class to keep partition key type as a string.
Contains additional methods to access the partition key for each dimension.
Creates a string representation of the partition key for each dimension, separated by a pipe (|).
Orders the dimensions by name, to ensure consistent string representation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DynamicPartitionsDefinition'>`class` dagster.DynamicPartitionsDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/dynamic.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicPartitionsDefinition" class="hash-link"></a></Link></dt>
<dd>
A partitions definition whose partition keys can be dynamically added and removed.
This is useful for cases where the set of partitions is not known at definition time,
but is instead determined at runtime.
Partitions can be added and removed using <cite>instance.add_dynamic_partitions</cite> and
<cite>instance.delete_dynamic_partition</cite> methods.
We recommended limiting partition counts for each asset to 100,000 partitions or fewer.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the partitions definition.
- <strong>partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>Optional</em><em>[</em><em>datetime</em><em>]</em><em>]</em><em>, </em><em>Union</em><em>[</em><em>Sequence</em><em>[</em><em>Partition</em><em>]</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> A function that returns the current set of partitions. This argument is deprecated and will be removed in 2.0.0.
Examples:
```python
fruits = DynamicPartitionsDefinition(name="fruits")
@sensor(job=my_job)
def my_sensor(context):
return SensorResult(
run_requests=[RunRequest(partition_key="apple")],
dynamic_partitions_requests=[fruits.build_add_request(["apple"])]
)
```
<dl>
<dt><Link class="anchor" id='dagster.DynamicPartitionsDefinition.get_partition_keys'>get_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/definition/dynamic.py#L134' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DynamicPartitionsDefinition.get_partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of strings representing the partition keys of the
PartitionsDefinition.
Parameters:
- <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – A datetime object representing the current time, only applicable to time-based partitions definitions.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore object that is responsible for fetching dynamic partitions. Required when the partitions definition is a DynamicPartitionsDefinition with a name defined. Users can pass the DagsterInstance fetched via <cite>context.instance</cite> to this argument.
Returns: Sequence[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionKeyRange'>`class` dagster.PartitionKeyRange <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partition_key_range.py#L7' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionKeyRange" class="hash-link"></a></Link></dt>
<dd>
Defines a range of partitions.
Parameters:
- <strong>start</strong> (<em>str</em>) – The starting partition key in the range (inclusive).
- <strong>end</strong> (<em>str</em>) – The ending partition key in the range (inclusive).
Examples:
```python
partitions_def = StaticPartitionsDefinition(["a", "b", "c", "d"])
partition_key_range = PartitionKeyRange(start="a", end="c") # Represents ["a", "b", "c"]
```
</dd>
</dl>
</div>
<div class="section" id="partitioned-schedules">
## Partitioned schedules
<dl>
<dt>dagster.build_schedule_from_partitioned_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_schedule.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Creates a schedule from a job that targets
time window-partitioned or statically-partitioned assets. The job can also be
multi-partitioned, as long as one of the partition dimensions is time-partitioned.
The schedule executes at the cadence specified by the time partitioning of the job or assets.
<strong>Example:</strong>
```python
######################################
# Job that targets partitioned assets
######################################
from dagster import (
DailyPartitionsDefinition,
asset,
build_schedule_from_partitioned_job,
define_asset_job,
Definitions,
)
@asset(partitions_def=DailyPartitionsDefinition(start_date="2020-01-01"))
def asset1():
...
asset1_job = define_asset_job("asset1_job", selection=[asset1])
# The created schedule will fire daily
asset1_job_schedule = build_schedule_from_partitioned_job(asset1_job)
Definitions(assets=[asset1], schedules=[asset1_job_schedule])
################
# Non-asset job
################
from dagster import DailyPartitionsDefinition, build_schedule_from_partitioned_job, jog
@job(partitions_def=DailyPartitionsDefinition(start_date="2020-01-01"))
def do_stuff_partitioned():
...
# The created schedule will fire daily
do_stuff_partitioned_schedule = build_schedule_from_partitioned_job(
do_stuff_partitioned,
)
Definitions(schedules=[do_stuff_partitioned_schedule])
```
</dd>
</dl>
</div>
<div class="section" id="partition-mapping">
## Partition mapping
<dl>
<dt><Link class="anchor" id='dagster.PartitionMapping'>`class` dagster.PartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/partition_mapping.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Defines a correspondence between the partitions in an asset and the partitions in an asset
that it depends on.
Overriding PartitionMapping outside of Dagster is not supported. The abstract methods of this
class may change at any time.
<dl>
<dt><Link class="anchor" id='dagster.PartitionMapping.get_downstream_partitions_for_partitions'>`abstractmethod` get_downstream_partitions_for_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/partition_mapping.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionMapping.get_downstream_partitions_for_partitions" class="hash-link"></a></Link></dt>
<dd>
Returns the subset of partition keys in the downstream asset that use the data in the given
partition key subset of the upstream asset.
Parameters:
- <strong>upstream_partitions_subset</strong> (<em>Union</em><em>[</em>[*PartitionKeyRange*](#dagster.PartitionKeyRange)<em>, </em><em>PartitionsSubset</em><em>]</em>) – The subset of partition keys in the upstream asset.
- <strong>downstream_partitions_def</strong> ([*PartitionsDefinition*](#dagster.PartitionsDefinition)) – The partitions definition for the downstream asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionMapping.get_upstream_mapped_partitions_result_for_partitions'>`abstractmethod` get_upstream_mapped_partitions_result_for_partitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/partition_mapping.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionMapping.get_upstream_mapped_partitions_result_for_partitions" class="hash-link"></a></Link></dt>
<dd>
Returns a UpstreamPartitionsResult object containing the partition keys the downstream
partitions subset was mapped to in the upstream partitions definition.
Valid upstream partitions will be included in UpstreamPartitionsResult.partitions_subset.
Invalid upstream partitions will be included in UpstreamPartitionsResult.required_but_nonexistent_subset.
For example, if an upstream asset is time-partitioned and starts in June 2023, and the
downstream asset is time-partitioned and starts in May 2023, this function would return a
UpstreamPartitionsResult(PartitionsSubset(“2023-06-01”), required_but_nonexistent_subset=PartitionsSubset(“2023-05-01”))
when downstream_partitions_subset contains 2023-05-01 and 2023-06-01.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.TimeWindowPartitionMapping'>`class` dagster.TimeWindowPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/time_window.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.TimeWindowPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
The default mapping between two TimeWindowPartitionsDefinitions.
A partition in the downstream partitions definition is mapped to all partitions in the upstream
asset whose time windows overlap it.
This means that, if the upstream and downstream partitions definitions share the same time
period, then this mapping is essentially the identity partition mapping - plus conversion of
datetime formats.
If the upstream time period is coarser than the downstream time period, then each partition in
the downstream asset will map to a single (larger) upstream partition. E.g. if the downstream is
hourly and the upstream is daily, then each hourly partition in the downstream will map to the
daily partition in the upstream that contains that hour.
If the upstream time period is finer than the downstream time period, then each partition in the
downstream asset will map to multiple upstream partitions. E.g. if the downstream is daily and
the upstream is hourly, then each daily partition in the downstream asset will map to the 24
hourly partitions in the upstream that occur on that day.
Parameters:
- <strong>start_offset</strong> (<em>int</em>) – If not 0, then the starts of the upstream windows are shifted by this offset relative to the starts of the downstream windows. For example, if start_offset=-1 and end_offset=0, then the downstream partition “2022-07-04” would map to the upstream partitions “2022-07-03” and “2022-07-04”. If the upstream and downstream PartitionsDefinitions are different, then the offset is in the units of the downstream. Defaults to 0.
- <strong>end_offset</strong> (<em>int</em>) – If not 0, then the ends of the upstream windows are shifted by this offset relative to the ends of the downstream windows. For example, if start_offset=0 and end_offset=1, then the downstream partition “2022-07-04” would map to the upstream partitions “2022-07-04” and “2022-07-05”. If the upstream and downstream PartitionsDefinitions are different, then the offset is in the units of the downstream. Defaults to 0.
- <strong>allow_nonexistent_upstream_partitions</strong> (<em>bool</em>) – <span className="flag flag-info">beta</span> Defaults to false. If true, does not raise an error when mapped upstream partitions fall outside the start-end time window of the partitions def. For example, if the upstream partitions def starts on “2023-01-01” but the downstream starts on “2022-01-01”, setting this bool to true would return no partition keys when get_upstream_partitions_for_partitions is called with “2022-06-01”. When set to false, would raise an error.
Examples:
```python
from dagster import DailyPartitionsDefinition, TimeWindowPartitionMapping, AssetIn, asset
partitions_def = DailyPartitionsDefinition(start_date="2020-01-01")
@asset(partitions_def=partitions_def)
def asset1():
...
@asset(
partitions_def=partitions_def,
ins={
"asset1": AssetIn(
partition_mapping=TimeWindowPartitionMapping(start_offset=-1)
)
}
)
def asset2(asset1):
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.IdentityPartitionMapping'>`class` dagster.IdentityPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/identity.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.IdentityPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Expects that the upstream and downstream assets are partitioned in the same way, and maps
partitions in the downstream asset to the same partition key in the upstream asset.
```python
import dagster as dg
daily_partitions_def = dg.DailyPartitionsDefinition(start_date="2025-01-01")
alternating_daily_partitions_def = dg.TimeWindowPartitionsDefinition(
start="2025-01-01",
fmt="%Y-%m-%d",
cron_schedule="0 0 */2 * *",
)
@dg.asset(partitions_def=daily_partitions_def)
def asset_upstream(context: dg.AssetExecutionContext): ...
# Downstream asset will map to the upstream when it is the same day
@dg.asset(
partitions_def=alternating_daily_partitions_def,
deps=[
dg.AssetDep(
asset=asset_upstream, partition_mapping=dg.IdentityPartitionMapping()
)
],
)
def asset_downstream(context: dg.AssetExecutionContext): ...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AllPartitionMapping'>`class` dagster.AllPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/all.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AllPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Maps every partition in the downstream asset to every partition in the upstream asset.
Commonly used in the case when the downstream asset is not partitioned, in which the entire
downstream asset depends on all partitions of the upstream asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.LastPartitionMapping'>`class` dagster.LastPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/last.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.LastPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Maps all dependencies to the last partition in the upstream asset.
Commonly used in the case when the downstream asset is not partitioned, in which the entire
downstream asset depends on the last partition of the upstream asset.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.StaticPartitionMapping'>`class` dagster.StaticPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/static.py#L25' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.StaticPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Define an explicit correspondence between two StaticPartitionsDefinitions.
Parameters: <strong>downstream_partition_keys_by_upstream_partition_key</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em> | </em><em>Collection</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The single or multi-valued correspondence from upstream keys to downstream keys.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SpecificPartitionsPartitionMapping'>`class` dagster.SpecificPartitionsPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/specific_partitions.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SpecificPartitionsPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
Maps to a specific subset of partitions in the upstream asset.
Example:
```python
from dagster import SpecificPartitionsPartitionMapping, StaticPartitionsDefinition, asset
@asset(partitions_def=StaticPartitionsDefinition(["a", "b", "c"]))
def upstream():
...
@asset(
ins={
"upstream": AssetIn(partition_mapping=SpecificPartitionsPartitionMapping(["a"]))
}
)
def a_downstream(upstream):
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiToSingleDimensionPartitionMapping'>`class` dagster.MultiToSingleDimensionPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/multi/multi_to_single.py#L161' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiToSingleDimensionPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Defines a correspondence between an single-dimensional partitions definition
and a MultiPartitionsDefinition. The single-dimensional partitions definition must be
a dimension of the MultiPartitionsDefinition.
This class handles the case where the upstream asset is multipartitioned and the
downstream asset is single dimensional, and vice versa.
For a partition key X, this partition mapping assumes that any multi-partition key with
X in the selected dimension is a dependency.
Parameters: <strong>partition_dimension_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the partition dimension in the
MultiPartitionsDefinition that matches the single-dimension partitions definition.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiPartitionMapping'>`class` dagster.MultiPartitionMapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/mapping/multi/multi_to_multi.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiPartitionMapping" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Defines a correspondence between two MultiPartitionsDefinitions.
Accepts a mapping of upstream dimension name to downstream DimensionPartitionMapping, representing
the explicit correspondence between the upstream and downstream MultiPartitions dimensions
and the partition mapping used to calculate the downstream partitions.
Examples:
```python
weekly_abc = MultiPartitionsDefinition(
{
"abc": StaticPartitionsDefinition(["a", "b", "c"]),
"weekly": WeeklyPartitionsDefinition("2023-01-01"),
}
)
daily_123 = MultiPartitionsDefinition(
{
"123": StaticPartitionsDefinition(["1", "2", "3"]),
"daily": DailyPartitionsDefinition("2023-01-01"),
}
)
MultiPartitionMapping(
{
"abc": DimensionPartitionMapping(
dimension_name="123",
partition_mapping=StaticPartitionMapping({"a": "1", "b": "2", "c": "3"}),
),
"weekly": DimensionPartitionMapping(
dimension_name="daily",
partition_mapping=TimeWindowPartitionMapping(),
)
}
)
```
For upstream or downstream dimensions not explicitly defined in the mapping, Dagster will
assume an <cite>AllPartitionsMapping</cite>, meaning that all upstream partitions in those dimensions
will be mapped to all downstream partitions in those dimensions.
Examples:
```python
weekly_abc = MultiPartitionsDefinition(
{
"abc": StaticPartitionsDefinition(["a", "b", "c"]),
"daily": DailyPartitionsDefinition("2023-01-01"),
}
)
daily_123 = MultiPartitionsDefinition(
{
"123": StaticPartitionsDefinition(["1", "2", "3"]),
"daily": DailyPartitionsDefinition("2023-01-01"),
}
)
MultiPartitionMapping(
{
"daily": DimensionPartitionMapping(
dimension_name="daily",
partition_mapping=IdentityPartitionMapping(),
)
}
)
# Will map `daily_123` partition key {"123": "1", "daily": "2023-01-01"} to the upstream:
# {"abc": "a", "daily": "2023-01-01"}
# {"abc": "b", "daily": "2023-01-01"}
# {"abc": "c", "daily": "2023-01-01"}
```
Parameters: <strong>downstream_mappings_by_upstream_dimension</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>DimensionPartitionMapping</em><em>]</em>) – A
mapping that defines an explicit correspondence between one dimension of the upstream
MultiPartitionsDefinition and one dimension of the downstream MultiPartitionsDefinition.
Maps a string representing upstream dimension name to downstream DimensionPartitionMapping,
containing the downstream dimension name and partition mapping.
</dd>
</dl>
</div>
<div class="section" id="backfill-policy">
## Backfill policy
<dl>
<dt><Link class="anchor" id='dagster.BackfillPolicy'>`class` dagster.BackfillPolicy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/backfill_policy.py#L16' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.BackfillPolicy" class="hash-link"></a></Link></dt>
<dd>
A BackfillPolicy specifies how Dagster should attempt to backfill a partitioned asset.
There are two main kinds of backfill policies: single-run and multi-run.
An asset with a single-run backfill policy will take a single run to backfill all of its
partitions at once.
An asset with a multi-run backfill policy will take multiple runs to backfill all of its
partitions. Each run will backfill a subset of the partitions. The number of partitions to
backfill in each run is controlled by the <cite>max_partitions_per_run</cite> parameter.
For example:
- If an asset has 100 partitions, and the <cite>max_partitions_per_run</cite> is set to 10, then it will be backfilled in 10 runs; each run will backfill 10 partitions.
- If an asset has 100 partitions, and the <cite>max_partitions_per_run</cite> is set to 11, then it will be backfilled in 10 runs; the first 9 runs will backfill 11 partitions, and the last run will backfill the one remaining partition.
<strong>Warning:</strong>
Constructing an BackfillPolicy directly is not recommended as the API is subject to change.
BackfillPolicy.single_run() and BackfillPolicy.multi_run(max_partitions_per_run=x) are the
recommended APIs.
<dl>
<dt><Link class="anchor" id='dagster.BackfillPolicy.multi_run'>`static` multi_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/backfill_policy.py#L65' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.BackfillPolicy.multi_run" class="hash-link"></a></Link></dt>
<dd>
Creates a BackfillPolicy that executes the entire backfill in multiple runs.
Each run will backfill [max_partitions_per_run] number of partitions.
Parameters: <strong>max_partitions_per_run</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The maximum number of partitions in each run of
the multiple runs. Defaults to 1.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.BackfillPolicy.single_run'>`static` single_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/backfill_policy.py#L59' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.BackfillPolicy.single_run" class="hash-link"></a></Link></dt>
<dd>
Creates a BackfillPolicy that executes the entire backfill in a single run.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="partitioned-config">
## Partitioned config
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig'>`class` dagster.PartitionedConfig <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/partitioned_config.py#L237' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig" class="hash-link"></a></Link></dt>
<dd>
Defines a way of configuring a job where the job can be run on one of a discrete set of
partitions, and each partition corresponds to run configuration for the job.
Setting PartitionedConfig as the config for a job allows you to launch backfills for that job
and view the run history across partitions.
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.get_partition_keys'>get_partition_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/partitioned_config.py#L142' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.get_partition_keys" class="hash-link"></a></Link></dt>
<dd>
Returns a list of partition keys, representing the full set of partitions that
config can be applied to.
Parameters: <strong>current_time</strong> (<em>Optional</em><em>[</em><em>datetime</em><em>]</em>) – A datetime object representing the current time. Only
applicable to time-based partitions definitions.Returns: Sequence[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.partitions_def'>`property` partitions_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/partitioned_config.py#L86' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.partitions_def" class="hash-link"></a></Link></dt>
<dd>
The partitions definition associated with this PartitionedConfig.
Type: T_PartitionsDefinition
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.run_config_for_partition_fn'>`property` run_config_for_partition_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.run_config_for_partition_fn" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `run_config_for_partition_key_fn` instead..
:::
A function that accepts a partition
and returns a dictionary representing the config to attach to runs for that partition.
Deprecated as of 1.3.3.
Type: Optional[Callable[[Partition], Mapping[str, Any]]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.run_config_for_partition_key_fn'>`property` run_config_for_partition_key_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/partitioned_config.py#L109' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.run_config_for_partition_key_fn" class="hash-link"></a></Link></dt>
<dd>
A function that accepts a partition key
and returns a dictionary representing the config to attach to runs for that partition.
Type: Optional[Callable[[str], Union[[RunConfig](config.mdx#dagster.RunConfig), Mapping[str, Any]]]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.tags_for_partition_fn'>`property` tags_for_partition_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L119' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.tags_for_partition_fn" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Use `tags_for_partition_key_fn` instead..
:::
A function that
accepts a partition and returns a dictionary of tags to attach to runs for
that partition. Deprecated as of 1.3.3.
Type: Optional[Callable[[Partition], Mapping[str, str]]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PartitionedConfig.tags_for_partition_key_fn'>`property` tags_for_partition_key_fn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/partitioned_config.py#L131' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PartitionedConfig.tags_for_partition_key_fn" class="hash-link"></a></Link></dt>
<dd>
A function that
accepts a partition key and returns a dictionary of tags to attach to runs for
that partition.
Type: Optional[Callable[[str], Mapping[str, str]]]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.static_partitioned_config'>dagster.static_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/static.py#L13' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.static_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Creates a static partitioned config for a job.
The provided partition_keys is a static list of strings identifying the set of partitions. The
list of partitions is static, so while the run config returned by the decorated function may
change over time, the list of valid partition keys does not.
This has performance advantages over <cite>dynamic_partitioned_config</cite> in terms of loading different
partition views in the Dagster UI.
The decorated function takes in a partition key and returns a valid run config for a particular
target job.
Parameters:
- <strong>partition_keys</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – A list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> A function that accepts a partition key and returns a dictionary of tags to attach to runs for that partition.
- <strong>tags_for_partition_key_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – A function that accepts a partition key and returns a dictionary of tags to attach to runs for that partition.
Returns: PartitionedConfig
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.dynamic_partitioned_config'>dagster.dynamic_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/dynamic.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.dynamic_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Creates a dynamic partitioned config for a job.
The provided partition_fn returns a list of strings identifying the set of partitions, given
an optional datetime argument (representing the current time). The list of partitions returned
may change over time.
The decorated function takes in a partition key and returns a valid run config for a particular
target job.
Parameters:
- <strong>partition_fn</strong> (<em>Callable</em><em>[</em><em>[</em><em>datetime.datetime</em><em>]</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A function that generates a list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> A function that accepts a partition key and returns a dictionary of tags to attach to runs for that partition.
Returns: PartitionedConfig
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.hourly_partitioned_config'>dagster.hourly_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/time_window.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.hourly_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Defines run config over a set of hourly partitions.
The decorated function should accept a start datetime and end datetime, which represent the date
partition the config should delineate.
The decorated function should return a run config dictionary.
The resulting object created by this decorator can be provided to the config argument of a Job.
The first partition in the set will start at the start_date at midnight. The last partition in
the set will end before the current time, unless the end_offset argument is set to a positive
number. If minute_offset is provided, the start and end times of each partition will be
minute_offset past the hour.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – A function that accepts a partition time window and returns a dictionary of tags to attach to runs for that partition.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
from datetime import datetime
from dagster import hourly_partitioned_config
@hourly_partitioned_config(start_date=datetime(2022, 3, 12))
def my_hourly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d %H:%M"), "end": end.strftime("%Y-%m-%d %H:%M")}
# creates partitions (2022-03-12-00:00, 2022-03-12-01:00), (2022-03-12-01:00, 2022-03-12-02:00), ...
@hourly_partitioned_config(start_date=datetime(2022, 3, 12), minute_offset=15)
def my_offset_hourly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d %H:%M"), "end": end.strftime("%Y-%m-%d %H:%M")}
# creates partitions (2022-03-12-00:15, 2022-03-12-01:15), (2022-03-12-01:15, 2022-03-12-02:15), ...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.daily_partitioned_config'>dagster.daily_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/time_window.py#L134' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.daily_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Defines run config over a set of daily partitions.
The decorated function should accept a start datetime and end datetime, which represent the bounds
of the date partition the config should delineate.
The decorated function should return a run config dictionary.
The resulting object created by this decorator can be provided to the config argument of a Job.
The first partition in the set will start at the start_date at midnight. The last partition in
the set will end before the current time, unless the end_offset argument is set to a positive
number. If minute_offset and/or hour_offset are used, the start and end times of each partition
will be hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – A function that accepts a partition time window and returns a dictionary of tags to attach to runs for that partition.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
from datetime import datetime
from dagster import daily_partitioned_config
@daily_partitioned_config(start_date="2022-03-12")
def my_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-03-12-00:00, 2022-03-13-00:00), (2022-03-13-00:00, 2022-03-14-00:00), ...
@daily_partitioned_config(start_date="2022-03-12", minute_offset=15, hour_offset=16)
def my_offset_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-03-12-16:15, 2022-03-13-16:15), (2022-03-13-16:15, 2022-03-14-16:15), ...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.weekly_partitioned_config'>dagster.weekly_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/time_window.py#L226' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.weekly_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Defines run config over a set of weekly partitions.
The decorated function should accept a start datetime and end datetime, which represent the date
partition the config should delineate.
The decorated function should return a run config dictionary.
The resulting object created by this decorator can be provided to the config argument of a Job.
The first partition in the set will start at the start_date. The last partition in the set will
end before the current time, unless the end_offset argument is set to a positive number. If
day_offset is provided, the start and end date of each partition will be day of the week
corresponding to day_offset (0 indexed with Sunday as the start of the week). If
minute_offset and/or hour_offset are used, the start and end times of each partition will be
hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions will Sunday at midnight following start_date. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>day_offset</strong> (<em>int</em>) – Day of the week to “split” the partition. Defaults to 0 (Sunday).
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – A function that accepts a partition time window and returns a dictionary of tags to attach to runs for that partition.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
from datetime import datetime
from dagster import weekly_partitioned_config
@weekly_partitioned_config(start_date="2022-03-12")
def my_weekly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-03-13-00:00, 2022-03-20-00:00), (2022-03-20-00:00, 2022-03-27-00:00), ...
@weekly_partitioned_config(start_date="2022-03-12", minute_offset=15, hour_offset=3, day_offset=6)
def my_offset_weekly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-03-12-03:15, 2022-03-19-03:15), (2022-03-19-03:15, 2022-03-26-03:15), ...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.monthly_partitioned_config'>dagster.monthly_partitioned_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_config/time_window.py#L323' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.monthly_partitioned_config" class="hash-link"></a></Link></dt>
<dd>
Defines run config over a set of monthly partitions.
The decorated function should accept a start datetime and end datetime, which represent the date
partition the config should delineate.
The decorated function should return a run config dictionary.
The resulting object created by this decorator can be provided to the config argument of a Job.
The first partition in the set will start at midnight on the soonest first of the month after
start_date. The last partition in the set will end before the current time, unless the
end_offset argument is set to a positive number. If day_offset is provided, the start and end
date of each partition will be day_offset. If minute_offset and/or hour_offset are used, the
start and end times of each partition will be hour_offset:minute_offset of each day.
Parameters:
- <strong>start_date</strong> (<em>Union</em><em>[</em><em>datetime.datetime</em><em>, </em><em>str</em><em>]</em>) – The first date in the set of partitions will be midnight the soonest first of the month following start_date. Can provide in either a datetime or string format.
- <strong>minute_offset</strong> (<em>int</em>) – Number of minutes past the hour to “split” the partition. Defaults to 0.
- <strong>hour_offset</strong> (<em>int</em>) – Number of hours past 00:00 to “split” the partition. Defaults to 0.
- <strong>day_offset</strong> (<em>int</em>) – Day of the month to “split” the partition. Defaults to 1.
- <strong>timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>fmt</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The date format to use. Defaults to <cite>%Y-%m-%d</cite>.
- <strong>end_offset</strong> (<em>int</em>) – Extends the partition set by a number of partitions equal to the value passed. If end_offset is 0 (the default), the last partition ends before the current time. If end_offset is 1, the second-to-last partition ends before the current time, and so on.
- <strong>tags_for_partition_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – A function that accepts a partition time window and returns a dictionary of tags to attach to runs for that partition.
- <strong>exclusions</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>datetime</em><em>]</em><em>]</em><em>]</em>) – Specifies a sequence of cron strings or datetime objects that should be excluded from the partition set. Every tick of the cron schedule that matches an excluded datetime or matches the tick of an excluded cron string will be excluded from the partition set.
```python
from datetime import datetime
from dagster import monthly_partitioned_config
@monthly_partitioned_config(start_date="2022-03-12")
def my_monthly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-04-01-00:00, 2022-05-01-00:00), (2022-05-01-00:00, 2022-06-01-00:00), ...
@monthly_partitioned_config(start_date="2022-03-12", minute_offset=15, hour_offset=3, day_offset=5)
def my_offset_monthly_partitioned_config(start: datetime, end: datetime):
return {"start": start.strftime("%Y-%m-%d"), "end": end.strftime("%Y-%m-%d")}
# creates partitions (2022-04-05-03:15, 2022-05-05-03:15), (2022-05-05-03:15, 2022-06-05-03:15), ...
```
</dd>
</dl>
</div>
<div class="section" id="partition-loading-context">
## Partition loading context
<dl>
<dt><Link class="anchor" id='dagster.partition_loading_context'>dagster.partition_loading_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/context.py#L87' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.partition_loading_context" class="hash-link"></a></Link></dt>
<dd>
Context manager for setting the current PartitionLoadingContext, which controls how PartitionsDefinitions,
PartitionMappings, and PartitionSubsets are loaded. This contextmanager is additive, meaning if effective_dt
or dynamic_partitions_store are not provided, the value from the previous PartitionLoadingContext is used if
it exists.
Parameters:
- <strong>effective_dt</strong> (<em>Optional</em><em>[</em><em>datetime.datetime</em><em>]</em>) – The effective time for the partition loading.
- <strong>dynamic_partitions_store</strong> (<em>Optional</em><em>[</em><em>DynamicPartitionsStore</em><em>]</em>) – The DynamicPartitionsStore backing the partition loading.
- <strong>new_ctx</strong> (<em>Optional</em><em>[</em><em>PartitionLoadingContext</em><em>]</em>) – A new PartitionLoadingContext which will override the current one.
Examples:
```python
import dagster as dg
import datetime
partitions_def = dg.DailyPartitionsDefinition(start_date="2021-01-01")
with dg.partition_loading_context(effective_dt=datetime.datetime(2021, 1, 2)):
assert partitions_def.get_last_partition_key() == "2021-01-01"
with dg.partition_loading_context(effective_dt=datetime.datetime(2021, 1, 3)):
assert partitions_def.get_last_partition_key() == "2021-01-02"
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster pipes'
sidebar_position: 1000
title_meta: 'dagster pipes API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster pipes Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pipes">
# Dagster Pipes
[Dagster Pipes](https://docs.dagster.io/integrations/external-pipelines) is a toolkit for building integrations between Dagster and external execution environments. This reference outlines the APIs included with the `dagster` library, which should be used in the orchestration environment.
For a detailed look at the Pipes process, including how to customize it, see [Dagster Pipes details and customization](https://docs.dagster.io/integrations/external-pipelines/dagster-pipes-details-and-customization).
<strong>Looking to write code in an external process?</strong> Refer to the API reference for the separately-installed [dagster-pipes](https://docs.dagster.io/integrations/libraries/pipes) library.
<div class="section" id="sessions">
## Sessions
<dl>
<dt><Link class="anchor" id='dagster.PipesSession'>`class` dagster.PipesSession <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L261' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession" class="hash-link"></a></Link></dt>
<dd>
Object representing a pipes session.
A pipes session is defined by a pair of [`PipesContextInjector`](#dagster.PipesContextInjector) and
[`PipesMessageReader`](#dagster.PipesMessageReader) objects. At the opening of the session, the context injector
writes context data to an externally accessible location, and the message reader starts
monitoring an externally accessible location. These locations are encoded in parameters stored
on a <cite>PipesSession</cite> object.
During the session, an external process should be started and the parameters injected into its
environment. The typical way to do this is to call [`PipesSession.get_bootstrap_env_vars()`](#dagster.PipesSession.get_bootstrap_env_vars)
and pass the result as environment variables.
During execution, results (e.g. asset materializations) are reported by the external process and
buffered on the <cite>PipesSession</cite> object. The buffer can periodically be cleared and yielded to
Dagster machinery by calling <cite>yield from PipesSession.get_results()</cite>.
When the external process exits, the session can be closed. Closing consists of handling any
unprocessed messages written by the external process and cleaning up any resources used for
context injection and message reading.
Parameters:
- <strong>context_data</strong> (<em>PipesContextData</em>) – The context for the executing op/asset.
- <strong>message_handler</strong> ([*PipesMessageHandler*](#dagster.PipesMessageHandler)) – The message handler to use for processing messages
- <strong>context_injector_params</strong> (<em>PipesParams</em>) – Parameters yielded by the context injector, indicating the location from which the external process should load context data.
- <strong>message_reader_params</strong> (<em>PipesParams</em>) – Parameters yielded by the message reader, indicating the location to which the external process should write messages.
- <strong>created_at</strong> (<em>datetime</em>) – The time at which the session was created. Useful as cutoff for reading logs.
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_bootstrap_cli_arguments'>get_bootstrap_cli_arguments <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L322' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_bootstrap_cli_arguments" class="hash-link"></a></Link></dt>
<dd>
Encode context injector and message reader params as CLI arguments.
Passing CLI arguments is an alternative way to expose the pipes I/O parameters to a pipes process.
Using environment variables should be preferred when possible.
Returns: CLI arguments pass to the external process. The values are
serialized as json, compressed with zlib, and then base64-encoded.Return type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_bootstrap_env_vars'>get_bootstrap_env_vars <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L306' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_bootstrap_env_vars" class="hash-link"></a></Link></dt>
<dd>
Encode context injector and message reader params as environment variables.
Passing environment variables is the typical way to expose the pipes I/O parameters
to a pipes process.
Returns: Environment variables to pass to the external process. The values are
serialized as json, compressed with gzip, and then base-64-encoded.Return type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_bootstrap_params'>get_bootstrap_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L383' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_bootstrap_params" class="hash-link"></a></Link></dt>
<dd>
Get the params necessary to bootstrap a launched pipes process. These parameters are typically
are as environment variable. See <cite>get_bootstrap_env_vars</cite>. It is the context injector’s
responsibility to decide how to pass these parameters to the external environment.
Returns: Parameters to pass to the external process and their corresponding
values that must be passed by the context injector.Return type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_custom_messages'>get_custom_messages <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_custom_messages" class="hash-link"></a></Link></dt>
<dd>
Get the sequence of deserialized JSON data that was reported from the external process using
<cite>report_custom_message</cite>.
Returns: Sequence[Any]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_per_writer_bootstrap_cli_arguments'>get_per_writer_bootstrap_cli_arguments <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L355' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_per_writer_bootstrap_cli_arguments" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Per-writer analog of <cite>get_bootstrap_cli_arguments</cite>. See
[`get_per_writer_bootstrap_env_vars()`](#dagster.PipesSession.get_per_writer_bootstrap_env_vars) for semantics.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_per_writer_bootstrap_env_vars'>get_per_writer_bootstrap_env_vars <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L338' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_per_writer_bootstrap_env_vars" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Encode bootstrap params for each external writer as environment variables.
Most sessions are single-writer and this returns a one-element sequence equivalent to
<cite>[get_bootstrap_env_vars()]</cite>. For sessions using [`PipesCompositeMessageReader`](#dagster.PipesCompositeMessageReader),
this returns N per-writer env var mappings, one per underlying reader. Callers that
launch multiple external processes that independently report back into a single
Dagster session (e.g. Databricks multi-task jobs) should pass each writer’s mapping
to its corresponding external process.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_reported_results'>get_reported_results <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_reported_results" class="hash-link"></a></Link></dt>
<dd>
`PipesExecutionResult` objects only explicitly received from the external process.
Returns: Result reported by external process.Return type: Sequence[PipesExecutionResult]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSession.get_results'>get_results <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L398' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSession.get_results" class="hash-link"></a></Link></dt>
<dd>
`PipesExecutionResult` objects reported from the external process,
potentially modified by Pipes.
Parameters:
- <strong>implicit_materializations</strong> (<em>bool</em>) – Create MaterializeResults for expected assets even was nothing is reported from the external process.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*MetadataValue*](metadata.mdx#dagster.MetadataValue)<em>]</em><em>]</em>) – Arbitrary metadata that will be attached to all results generated by the invocation. Useful for attaching information to asset materializations and checks that is available via the external process launch API but not in the external process itself (e.g. a job_id param returned by the launch API call).
Returns: Result reported by external process.Return type: Sequence[PipesExecutionResult]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesLaunchedData'>`class` dagster.PipesLaunchedData <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesLaunchedData" class="hash-link"></a></Link></dt>
<dd>
Payload generated in the orchestration process after external process startup
containing arbitrary information about the external process.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.open_pipes_session'>dagster.open_pipes_session <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L859' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.open_pipes_session" class="hash-link"></a></Link></dt>
<dd>
Context manager that opens and closes a pipes session.
This context manager should be used to wrap the launch of an external process using the pipe
protocol to report results back to Dagster. The yielded [`PipesSession`](#dagster.PipesSession) should be used
to (a) obtain the environment variables that need to be provided to the external process; (b)
access results streamed back from the external process.
This method is an alternative to [`PipesClient`](#dagster.PipesClient) subclasses for users who want more
control over how pipes processes are launched. When using <cite>open_pipes_session</cite>, it is the user’s
responsibility to inject the message reader and context injector parameters available on the
yielded <cite>PipesSession</cite> and pass them to the appropriate API when launching the external process.
Typically these parameters should be set as environment variables.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context for the current op/asset execution.
- <strong>context_injector</strong> ([*PipesContextInjector*](#dagster.PipesContextInjector)) – The context injector to use to inject context into the external process.
- <strong>message_reader</strong> ([*PipesMessageReader*](#dagster.PipesMessageReader)) – The message reader to use to read messages from the external process.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>PipesExtras</em><em>]</em>) – Optional extras to pass to the external process via the injected context.
Yields: <em>PipesSession</em> – Interface for interacting with the external process.
```python
import subprocess
from dagster import open_pipes_session
extras = {"foo": "bar"}
@asset
def ext_asset(context: AssetExecutionContext):
with open_pipes_session(
context=context,
extras={"foo": "bar"},
context_injector=PipesTempFileContextInjector(),
message_reader=PipesTempFileMessageReader(),
) as pipes_session:
subprocess.Popen(
["/bin/python", "/path/to/script.py"],
env={**pipes_session.get_bootstrap_env_vars()}
)
while process.poll() is None:
yield from pipes_session.get_results()
yield from pipes_session.get_results()
```
</dd>
</dl>
</div>
<div class="section" id="clients">
## Clients
<dl>
<dt><Link class="anchor" id='dagster.PipesClient'>`class` dagster.PipesClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/client.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesClient" class="hash-link"></a></Link></dt>
<dd>
Pipes client base class.
Pipes clients for specific external environments should subclass this.
<dl>
<dt><Link class="anchor" id='dagster.PipesClient.run'>`abstractmethod` run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/client.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesClient.run" class="hash-link"></a></Link></dt>
<dd>
Synchronously execute an external process with the pipes protocol. Derived
clients must have <cite>context</cite> and <cite>extras</cite> arguments, but also can add arbitrary
arguments that are appropriate for their own implementation.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context from the executing op/asset.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>PipesExtras</em><em>]</em>) – Arbitrary data to pass to the external environment.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesClientCompletedInvocation'>`class` dagster.PipesClientCompletedInvocation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/client.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesClientCompletedInvocation" class="hash-link"></a></Link></dt>
<dd>
A wrapper for the results of a pipes client invocation, typically returned from <cite>PipesClient.run</cite>.
Parameters:
- <strong>session</strong> ([*PipesSession*](#dagster.PipesSession)) – The Pipes session that was used to run the external process.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>RawMetadataMapping</em><em>]</em>) – Arbitrary metadata that will be attached to all results generated by the invocation. Useful for attaching information to asset materializations and checks that is available via the external process launch API but not in the external process itself (e.g. a job_id param returned by the launch API call).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesSubprocessClient'>`class` dagster.PipesSubprocessClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/subprocess.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSubprocessClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client that runs a subprocess with the given command and environment.
By default parameters are injected via environment variables. Context is passed via
a temp file, and structured messages are read from from a temp file.
Parameters:
- <strong>env</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – An optional dict of environment variables to pass to the subprocess.
- <strong>cwd</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Working directory in which to launch the subprocess command.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the subprocess. Defaults to [`PipesTempFileContextInjector`](#dagster.PipesTempFileContextInjector).
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the subprocess. Defaults to [`PipesTempFileMessageReader`](#dagster.PipesTempFileMessageReader).
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to send a SIGINT signal to the subprocess if the orchestration process is interrupted or canceled. Defaults to True.
- <strong>forward_stdio</strong> (<em>bool</em>) – Whether to forward stdout and stderr from the subprocess to the orchestration process. Defaults to True.
- <strong>termination_timeout_seconds</strong> (<em>float</em>) – How long to wait after forwarding termination for the subprocess to exit. Defaults to 20.
<dl>
<dt><Link class="anchor" id='dagster.PipesSubprocessClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/subprocess.py#L88' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesSubprocessClient.run" class="hash-link"></a></Link></dt>
<dd>
Synchronously execute a subprocess with in a pipes session.
Parameters:
- <strong>command</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The command to run. Will be passed to <cite>subprocess.Popen()</cite>.
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context from the executing op or asset.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>PipesExtras</em><em>]</em>) – An optional dict of extra parameters to pass to the subprocess.
- <strong>env</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – An optional dict of environment variables to pass to the subprocess.
- <strong>cwd</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Working directory in which to launch the subprocess command.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="advanced">
## Advanced
Most Pipes users won’t need to use the APIs in the following sections unless they are customizing the Pipes protocol.
Refer to the [Dagster Pipes details and customization guide](https://docs.dagster.io/integrations/external-pipelines/dagster-pipes-details-and-customization) for more information.
<div class="section" id="context-injectors">
### Context injectors
Context injectors write context payloads to an externally accessible location and yield a set of parameters encoding the location for inclusion in the bootstrap payload.
<dl>
<dt><Link class="anchor" id='dagster.PipesContextInjector'>`class` dagster.PipesContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/client.py#L138' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesContextInjector" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesEnvContextInjector'>`class` dagster.PipesEnvContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L110' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesEnvContextInjector" class="hash-link"></a></Link></dt>
<dd>
Context injector that injects context data into the external process by injecting it directly into the external process environment.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesFileContextInjector'>`class` dagster.PipesFileContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L48' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesFileContextInjector" class="hash-link"></a></Link></dt>
<dd>
Context injector that injects context data into the external process by writing it to a
specified file.
Parameters: <strong>path</strong> (<em>str</em>) – The path of a file to which to write context data. The file will be deleted on
close of the pipes session.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesTempFileContextInjector'>`class` dagster.PipesTempFileContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesTempFileContextInjector" class="hash-link"></a></Link></dt>
<dd>
Context injector that injects context data into the external process by writing it to an
automatically-generated temporary file.
</dd>
</dl>
</div>
<div class="section" id="message-readers">
### Message readers
Message readers read messages (and optionally log files) from an externally accessible location and yield a set of parameters encoding the location in the bootstrap payload.
<dl>
<dt><Link class="anchor" id='dagster.PipesMessageReader'>`class` dagster.PipesMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/client.py#L166' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesMessageReader" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesBlobStoreMessageReader'>`class` dagster.PipesBlobStoreMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L668' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesBlobStoreMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads a sequence of message chunks written by an external process into a
blob store such as S3, Azure blob storage, or GCS.
The reader maintains a counter, starting at 1, that is synchronized with a message writer in
some pipes process. The reader starts a thread that periodically attempts to read a chunk
indexed by the counter at some location expected to be written by the pipes process. The chunk
should be a file with each line corresponding to a JSON-encoded pipes message. When a chunk is
successfully read, the messages are processed and the counter is incremented. The
`PipesBlobStoreMessageWriter` on the other end is expected to similarly increment a
counter (starting from 1) on successful write, keeping counters on the read and write end in
sync.
If <cite>log_readers</cite> is passed, the message reader will start the passed log readers when the
<cite>opened</cite> message is received from the external process.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a chunk
- <strong>log_readers</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>PipesLogReader</em><em>]</em><em>]</em>) – A set of log readers to use to read logs.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesCompositeMessageReader'>`class` dagster.PipesCompositeMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L269' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesCompositeMessageReader" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A message reader that composes multiple underlying readers, one per external writer.
Use this when a single Dagster invocation launches multiple external processes that each
open their own Pipes session (e.g. Databricks multi-task jobs). Each underlying reader
owns its own independent destination (S3 prefix, DBFS/UC Volumes tempdir, local file, etc.)
so writers never collide on chunk filenames. Messages from all underlying readers are
funneled into the same [`PipesMessageHandler`](#dagster.PipesMessageHandler), which tracks aggregate
<cite>opened</cite>/<cite>closed</cite> state via its <cite>expected_writer_count</cite>-aware counters.
The <cite>read_messages</cite> method yields a params dict of the form
`\{PIPES_COMPOSITE_WRITERS_KEY: [\<per-writer params>, ...]}`. Session helpers
[`PipesSession.get_per_writer_bootstrap_env_vars()`](#dagster.PipesSession.get_per_writer_bootstrap_env_vars) and
[`PipesSession.get_per_writer_bootstrap_cli_arguments()`](#dagster.PipesSession.get_per_writer_bootstrap_cli_arguments) unpack that list; callers
are responsible for pairing each external process with its corresponding writer params.
Parameters: <strong>readers</strong> (<em>Sequence</em><em>[</em>[*PipesMessageReader*](#dagster.PipesMessageReader)<em>]</em>) – The underlying readers, one per expected
external writer. Each reader must itself be single-writer; composing composite
readers is not supported.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesFileMessageReader'>`class` dagster.PipesFileMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesFileMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by tailing a specified file.
Parameters:
- <strong>path</strong> (<em>str</em>) – The path of the file to which messages will be written. The file will be deleted on close of the pipes session.
- <strong>include_stdio_in_messages</strong> (<em>bool</em>) – Whether to include stdout/stderr logs in the messages produced by the message writer in the external process.
- <strong>cleanup_file</strong> (<em>bool</em>) – Whether to delete the file on close of the pipes session.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesTempFileMessageReader'>`class` dagster.PipesTempFileMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/utils.py#L210' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesTempFileMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by tailing an automatically-generated temporary file.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PipesMessageHandler'>`class` dagster.PipesMessageHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/pipes/context.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PipesMessageHandler" class="hash-link"></a></Link></dt>
<dd>
Class to process `PipesMessage` objects received from a pipes process.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context for the executing op/asset.
- <strong>message_reader</strong> ([*PipesMessageReader*](#dagster.PipesMessageReader)) – The message reader used to read messages from the external process.
</dd>
</dl>
</div></div></div>
---
---
title: 'repositories'
sidebar_position: 1000
title_meta: 'repositories API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'repositories Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="repositories">
# Repositories
<dl>
<dt><Link class="anchor" id='dagster.repository'>dagster.repository RepositoryDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/repository_decorator.py#L250' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.repository" class="hash-link"></a></Link></dt>
<dd>
Create a repository from the decorated function.
In most cases, [`Definitions`](definitions.mdx#dagster.Definitions) should be used instead.
The decorated function should take no arguments and its return value should one of:
1. `List[Union[JobDefinition, ScheduleDefinition, SensorDefinition]]`.
Use this form when you have no need to lazy load jobs or other definitions. This is the
typical use case.
2. A dict of the form:
```python
{
'jobs': Dict[str, Callable[[], JobDefinition]],
'schedules': Dict[str, Callable[[], ScheduleDefinition]],
'sensors': Dict[str, Callable[[], SensorDefinition]]
}
```
This form is intended to allow definitions to be created lazily when accessed by name,
which can be helpful for performance when there are many definitions in a repository, or
when constructing the definitions is costly.
3. A [`RepositoryData`](#dagster.RepositoryData). Return this object if you need fine-grained
control over the construction and indexing of definitions within the repository, e.g., to
create definitions dynamically from .yaml files in a directory.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the repository. Defaults to the name of the decorated function.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string description of the repository.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>RawMetadataValue</em><em>]</em><em>]</em>) – Arbitrary metadata for the repository. Not displayed in the UI but accessible on RepositoryDefinition at runtime.
Example:
```python
######################################################################
# A simple repository using the first form of the decorated function
######################################################################
@op(config_schema={n: Field(Int)})
def return_n(context):
return context.op_config['n']
@job
def simple_job():
return_n()
@job
def some_job():
...
@sensor(job=some_job)
def some_sensor():
if foo():
yield RunRequest(
run_key=...,
run_config={
'ops': {'return_n': {'config': {'n': bar()}}}
}
)
@job
def my_job():
...
my_schedule = ScheduleDefinition(cron_schedule="0 0 * * *", job=my_job)
@repository
def simple_repository():
return [simple_job, some_sensor, my_schedule]
######################################################################
# A simple repository using the first form of the decorated function
# and custom metadata that will be displayed in the UI
######################################################################
...
@repository(
name='my_repo',
metadata={
'team': 'Team A',
'repository_version': '1.2.3',
'environment': 'production',
})
def simple_repository():
return [simple_job, some_sensor, my_schedule]
######################################################################
# A lazy-loaded repository
######################################################################
def make_expensive_job():
@job
def expensive_job():
for i in range(10000):
return_n.alias(f'return_n_{i}')()
return expensive_job
def make_expensive_schedule():
@job
def other_expensive_job():
for i in range(11000):
return_n.alias(f'my_return_n_{i}')()
return ScheduleDefinition(cron_schedule="0 0 * * *", job=other_expensive_job)
@repository
def lazy_loaded_repository():
return {
'jobs': {'expensive_job': make_expensive_job},
'schedules': {'expensive_schedule': make_expensive_schedule}
}
######################################################################
# A complex repository that lazily constructs jobs from a directory
# of files in a bespoke YAML format
######################################################################
class ComplexRepositoryData(RepositoryData):
def __init__(self, yaml_directory):
self._yaml_directory = yaml_directory
def get_all_jobs(self):
return [
self._construct_job_def_from_yaml_file(
self._yaml_file_for_job_name(file_name)
)
for file_name in os.listdir(self._yaml_directory)
]
...
@repository
def complex_repository():
return ComplexRepositoryData('some_directory')
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition'>`class` dagster.RepositoryDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L448' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition" class="hash-link"></a></Link></dt>
<dd>
Define a repository that contains a group of definitions.
Users should typically not create objects of this class directly. Instead, use the
`@repository()` decorator.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the repository.
- <strong>repository_data</strong> ([*RepositoryData*](#dagster.RepositoryData)) – Contains the definitions making up the repository.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string description of the repository.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>MetadataMapping</em><em>]</em>) – Arbitrary metadata for the repository. Not displayed in the UI but accessible on RepositoryDefinition at runtime.
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.get_all_jobs'>get_all_jobs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L247' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.get_all_jobs" class="hash-link"></a></Link></dt>
<dd>
Return all jobs in the repository as a list.
Note that this will construct any job in the lazily evaluated dictionary that has
not yet been constructed.
Returns: All jobs in the repository.Return type: List[[JobDefinition](jobs.mdx#dagster.JobDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.get_asset_value_loader'>get_asset_value_loader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L408' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.get_asset_value_loader" class="hash-link"></a></Link></dt>
<dd>
Returns an object that can load the contents of assets as Python objects.
Invokes <cite>load_input</cite> on the [`IOManager`](io-managers.mdx#dagster.IOManager) associated with the assets. Avoids
spinning up resources separately for each asset.
Usage:
```python
with my_repo.get_asset_value_loader() as loader:
asset1 = loader.load_asset_value("asset1")
asset2 = loader.load_asset_value("asset2")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.get_job'>get_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L230' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.get_job" class="hash-link"></a></Link></dt>
<dd>
Get a job by name.
If this job is present in the lazily evaluated dictionary passed to the
constructor, but has not yet been constructed, only this job is constructed, and
will be cached for future calls.
Parameters: <strong>name</strong> (<em>str</em>) – Name of the job to retrieve.Returns: The job definition corresponding to
the given name.Return type: [JobDefinition](jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.get_schedule_def'>get_schedule_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L265' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.get_schedule_def" class="hash-link"></a></Link></dt>
<dd>
Get a schedule definition by name.
Parameters: <strong>name</strong> (<em>str</em>) – The name of the schedule.Returns: The schedule definition.Return type: [ScheduleDefinition](schedules-sensors.mdx#dagster.ScheduleDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.get_sensor_def'>get_sensor_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L288' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.get_sensor_def" class="hash-link"></a></Link></dt>
<dd>
Get a sensor definition by name.
Parameters: <strong>name</strong> (<em>str</em>) – The name of the sensor.Returns: The sensor definition.Return type: [SensorDefinition](schedules-sensors.mdx#dagster.SensorDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.has_job'>has_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L218' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.has_job" class="hash-link"></a></Link></dt>
<dd>
Check if a job with a given name is present in the repository.
Parameters: <strong>name</strong> (<em>str</em>) – The name of the job.Returns: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.has_schedule_def'>has_schedule_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L277' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.has_schedule_def" class="hash-link"></a></Link></dt>
<dd>
bool: Check if a schedule with a given name is present in the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.has_sensor_def'>has_sensor_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L300' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.has_sensor_def" class="hash-link"></a></Link></dt>
<dd>
bool: Check if a sensor with a given name is present in the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.load_asset_value'>load_asset_value <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L361' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.load_asset_value" class="hash-link"></a></Link></dt>
<dd>
Load the contents of an asset as a Python object.
Invokes <cite>load_input</cite> on the [`IOManager`](io-managers.mdx#dagster.IOManager) associated with the asset.
If you want to load the values of multiple assets, it’s more efficient to use
[`get_asset_value_loader()`](#dagster.RepositoryDefinition.get_asset_value_loader), which avoids spinning up
resources separately for each asset.
Parameters:
- <strong>asset_key</strong> (<em>Union</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em>) – The key of the asset to load.
- <strong>python_type</strong> (<em>Optional</em><em>[</em><em>Type</em><em>]</em>) – The python type to load the asset as. This is what will be returned inside <cite>load_input</cite> by <cite>context.dagster_type.typing_type</cite>.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The partition of the asset to load.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Input metadata to pass to the [`IOManager`](io-managers.mdx#dagster.IOManager) (is equivalent to setting the metadata argument in <cite>In</cite> or <cite>AssetIn</cite>).
- <strong>resource_config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – A dictionary of resource configurations to be passed to the [`IOManager`](io-managers.mdx#dagster.IOManager).
Returns: The contents of an asset as a Python object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.asset_checks_defs_by_key'>`property` asset_checks_defs_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L327' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.asset_checks_defs_by_key" class="hash-link"></a></Link></dt>
<dd>
The assets checks defined in the repository.
Type: Mapping[[AssetCheckKey](asset-checks.mdx#dagster.AssetCheckKey), [AssetChecksDefinition](asset-checks.mdx#dagster.AssetChecksDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.assets_defs_by_key'>`property` assets_defs_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L321' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.assets_defs_by_key" class="hash-link"></a></Link></dt>
<dd>
The assets definitions defined in the repository.
Type: Mapping[[AssetKey](assets.mdx#dagster.AssetKey), [AssetsDefinition](assets.mdx#dagster.AssetsDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L186' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.description" class="hash-link"></a></Link></dt>
<dd>
A human-readable description of the repository.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.job_names'>`property` job_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L206' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.job_names" class="hash-link"></a></Link></dt>
<dd>
Names of all jobs in the repository.
Type: List[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.metadata'>`property` metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L192' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.metadata" class="hash-link"></a></Link></dt>
<dd>
Arbitrary metadata for the repository.
Type: Optional[MetadataMapping]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L180' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.name" class="hash-link"></a></Link></dt>
<dd>
The name of the repository.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.schedule_defs'>`property` schedule_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L259' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.schedule_defs" class="hash-link"></a></Link></dt>
<dd>
All schedules in the repository.
Type: List[[ScheduleDefinition](schedules-sensors.mdx#dagster.ScheduleDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.sensor_defs'>`property` sensor_defs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L282' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.sensor_defs" class="hash-link"></a></Link></dt>
<dd>
All sensors in the repository.
Type: Sequence[[SensorDefinition](schedules-sensors.mdx#dagster.SensorDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryDefinition.source_assets_by_key'>`property` source_assets_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_definition.py#L305' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryDefinition.source_assets_by_key" class="hash-link"></a></Link></dt>
<dd>
The source assets defined in the repository.
Type: Mapping[[AssetKey](assets.mdx#dagster.AssetKey), [SourceAsset](assets.mdx#dagster.SourceAsset)]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData'>`class` dagster.RepositoryData <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData" class="hash-link"></a></Link></dt>
<dd>
Users should usually rely on the [`@repository`](#dagster.repository) decorator to create new
repositories, which will in turn call the static constructors on this class. However, users may
subclass [`RepositoryData`](#dagster.RepositoryData) for fine-grained control over access to and lazy creation
of repository members.
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_all_jobs'>`abstractmethod` get_all_jobs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L76' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_all_jobs" class="hash-link"></a></Link></dt>
<dd>
Return all jobs in the repository as a list.
Returns: All jobs in the repository.Return type: List[[JobDefinition](jobs.mdx#dagster.JobDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_all_schedules'>get_all_schedules <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L130' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_all_schedules" class="hash-link"></a></Link></dt>
<dd>
Return all schedules in the repository as a list.
Returns: All jobs in the repository.Return type: List[[ScheduleDefinition](schedules-sensors.mdx#dagster.ScheduleDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_all_sensors'>get_all_sensors <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L163' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_all_sensors" class="hash-link"></a></Link></dt>
<dd>
Sequence[SensorDefinition]: Return all sensors in the repository as a list.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_asset_checks_defs_by_key'>get_asset_checks_defs_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L207' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_asset_checks_defs_by_key" class="hash-link"></a></Link></dt>
<dd>
Mapping[AssetCheckKey, AssetChecksDefinition]: Get the asset checks definitions for the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_assets_defs_by_key'>get_assets_defs_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_assets_defs_by_key" class="hash-link"></a></Link></dt>
<dd>
Mapping[AssetKey, AssetsDefinition]: Get the asset definitions for the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_job'>get_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L106' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_job" class="hash-link"></a></Link></dt>
<dd>
Get a job by name.
Parameters: <strong>job_name</strong> (<em>str</em>) – Name of the job to retrieve.Returns: The job definition corresponding to the given name.Return type: [JobDefinition](jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_job_names'>get_job_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_job_names" class="hash-link"></a></Link></dt>
<dd>
Get the names of all jobs in the repository.
Returns: List[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_schedule'>get_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L139' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_schedule" class="hash-link"></a></Link></dt>
<dd>
Get a schedule by name.
Parameters: <strong>schedule_name</strong> (<em>str</em>) – name of the schedule to retrieve.Returns: The schedule definition corresponding to the given name.Return type: [ScheduleDefinition](schedules-sensors.mdx#dagster.ScheduleDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_schedule_names'>get_schedule_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L121' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_schedule_names" class="hash-link"></a></Link></dt>
<dd>
Get the names of all schedules in the repository.
Returns: List[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_sensor'>get_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L173' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_sensor" class="hash-link"></a></Link></dt>
<dd>
Get a sensor by name.
Parameters: <strong>sensor_name</strong> (<em>str</em>) – name of the sensor to retrieve.Returns: The sensor definition corresponding to the given name.Return type: [SensorDefinition](schedules-sensors.mdx#dagster.SensorDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_sensor_names'>get_sensor_names <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L168' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_sensor_names" class="hash-link"></a></Link></dt>
<dd>
Sequence[str]: Get the names of all sensors in the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.get_source_assets_by_key'>get_source_assets_by_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L197' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.get_source_assets_by_key" class="hash-link"></a></Link></dt>
<dd>
Mapping[AssetKey, SourceAsset]: Get the source assets for the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.has_job'>has_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.has_job" class="hash-link"></a></Link></dt>
<dd>
Check if a job with a given name is present in the repository.
Parameters: <strong>job_name</strong> (<em>str</em>) – The name of the job.Returns: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.has_schedule'>has_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L158' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.has_schedule" class="hash-link"></a></Link></dt>
<dd>
Check if a schedule with a given name is present in the repository.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositoryData.has_sensor'>has_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/repository_definition/repository_data.py#L192' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositoryData.has_sensor" class="hash-link"></a></Link></dt>
<dd>
Check if a sensor with a given name is present in the repository.
</dd>
</dl>
</dd>
</dl>
</div>
---
---
title: 'resources'
sidebar_position: 1000
title_meta: 'resources API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'resources Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="resources">
# Resources
<div class="section" id="pythonic-resource-system">
## Pythonic resource system
The following classes are used as part of the new [Pythonic resources system](https://docs.dagster.io/guides/build/external-resources).
<dl>
<dt><Link class="anchor" id='dagster.ConfigurableResource'>`class` dagster.ConfigurableResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/pythonic_config/resource.py#L596' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ConfigurableResource" class="hash-link"></a></Link></dt>
<dd>
Base class for Dagster resources that utilize structured config.
This class is a subclass of both [`ResourceDefinition`](#dagster.ResourceDefinition) and [`Config`](config.mdx#dagster.Config).
Example definition:
```python
class WriterResource(ConfigurableResource):
prefix: str
def output(self, text: str) -> None:
print(f"{self.prefix}{text}")
```
Example usage:
```python
@asset
def asset_that_uses_writer(writer: WriterResource):
writer.output("text")
defs = Definitions(
assets=[asset_that_uses_writer],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
You can optionally use this class to model configuration only and vend an object
of a different type for use at runtime. This is useful for those who wish to
have a separate object that manages configuration and a separate object at runtime. Or
where you want to directly use a third-party class that you do not control.
To do this you override the <cite>create_resource</cite> methods to return a different object.
```python
class WriterResource(ConfigurableResource):
prefix: str
def create_resource(self, context: InitResourceContext) -> Writer:
# Writer is pre-existing class defined else
return Writer(self.prefix)
```
Example usage:
```python
@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
writer.output("text")
defs = Definitions(
assets=[use_preexisting_writer_as_resource],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition'>`class` dagster.ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L247' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition" class="hash-link"></a></Link></dt>
<dd>
Core class for defining resources.
Resources are scoped ways to make external resources (like database connections) available to
ops and assets during job execution and to clean up after execution resolves.
If resource_fn yields once rather than returning (in the manner of functions decorable with
`@contextlib.contextmanager`) then the body of the
function after the yield will be run after execution resolves, allowing users to write their
own teardown/cleanup logic.
Depending on your executor, resources may be instantiated and cleaned up more than once in a
job execution.
Parameters:
- <strong>resource_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitResourceContext*](#dagster.InitResourceContext)<em>]</em><em>, </em><em>Any</em><em>]</em>) – User-provided function to instantiate the resource, which will be made available to executions keyed on the `context.resources` object.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The schema for the config. If set, Dagster will check that config provided for the resource matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the resource.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the resource.
- <strong>required_resource_keys</strong> – (Optional[Set[str]]) Keys for the resources required by this resource. A DagsterInvariantViolationError will be raised during initialization if dependencies are cyclic.
- <strong>version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) The version of the resource’s definition fn. Two wrapped resource functions should only have the same version if they produce the same resource definition when provided with the same inputs.
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.hardcoded_resource'>`static` hardcoded_resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L171' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.hardcoded_resource" class="hash-link"></a></Link></dt>
<dd>
A helper function that creates a `ResourceDefinition` with a hardcoded object.
Parameters:
- <strong>value</strong> (<em>Any</em>) – The value that will be accessible via context.resources.resource_name.
- <strong>description</strong> (<em>[</em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The description of the resource. Defaults to None.
Returns: A hardcoded resource.Return type: [[ResourceDefinition](#dagster.ResourceDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.mock_resource'>`static` mock_resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L193' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.mock_resource" class="hash-link"></a></Link></dt>
<dd>
A helper function that creates a `ResourceDefinition` which wraps a `mock.MagicMock`.
Parameters: <strong>description</strong> (<em>[</em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The description of the resource. Defaults to None.Returns:
A resource that creates the magic methods automatically and helps
you mock existing resources.
Return type: [[ResourceDefinition](#dagster.ResourceDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.none_resource'>`static` none_resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L158' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.none_resource" class="hash-link"></a></Link></dt>
<dd>
A helper function that returns a none resource.
Parameters: <strong>description</strong> (<em>[</em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The description of the resource. Defaults to None.Returns: A resource that does nothing.Return type: [[ResourceDefinition](#dagster.ResourceDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.string_resource'>`static` string_resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L211' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.string_resource" class="hash-link"></a></Link></dt>
<dd>
Creates a `ResourceDefinition` which takes in a single string as configuration
and returns this configured string to any ops or assets which depend on it.
Parameters: <strong>description</strong> (<em>[</em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The description of the string resource. Defaults to None.Returns:
A resource that takes in a single string as configuration and
returns that string.
Return type: [[ResourceDefinition](#dagster.ResourceDefinition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L129' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.description" class="hash-link"></a></Link></dt>
<dd>
A human-readable description of the resource.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L141' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
A set of the resource keys that this resource depends on. These keys will be made available
to the resource’s init context during execution, and the resource will not be instantiated
until all required resources are available.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ResourceDefinition.version'>`property` version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ResourceDefinition.version" class="hash-link"></a></Link></dt>
<dd>
A string which can be used to identify a particular code version of a resource definition.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext'>`class` dagster.InitResourceContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext" class="hash-link"></a></Link></dt>
<dd>
The context object available as the argument to the initialization function of a [`dagster.ResourceDefinition`](#dagster.ResourceDefinition).
Users should not instantiate this object directly. To construct an <cite>InitResourceContext</cite> for testing purposes, use [`dagster.build_init_resource_context()`](#dagster.build_init_resource_context).
Example:
```python
from dagster import resource, InitResourceContext
@resource
def the_resource(init_context: InitResourceContext):
init_context.log.info("Hello, world!")
```
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L75' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.instance" class="hash-link"></a></Link></dt>
<dd>
The Dagster instance configured for the current execution context.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.log" class="hash-link"></a></Link></dt>
<dd>
The Dagster log manager configured for the current execution context.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.log_manager'>`property` log_manager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L104' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.log_manager" class="hash-link"></a></Link></dt>
<dd>
The log manager for this run of the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.resource_config'>`property` resource_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.resource_config" class="hash-link"></a></Link></dt>
<dd>
The configuration data provided by the run config. The schema
for this data is defined by the `config_field` argument to
[`ResourceDefinition`](#dagster.ResourceDefinition).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.resource_def'>`property` resource_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.resource_def" class="hash-link"></a></Link></dt>
<dd>
The definition of the resource currently being constructed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.resources" class="hash-link"></a></Link></dt>
<dd>
The resources that are available to the resource that we are initializing.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.InitResourceContext.run'>`property` run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L81' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.InitResourceContext.run" class="hash-link"></a></Link></dt>
<dd>
The dagster run to use. When initializing resources outside of execution context, this will be None.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.make_values_resource'>dagster.make_values_resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L415' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.make_values_resource" class="hash-link"></a></Link></dt>
<dd>
A helper function that creates a `ResourceDefinition` to take in user-defined values.
>
This is useful for sharing values between ops.
Parameters: <strong>**kwargs</strong> – Arbitrary keyword arguments that will be passed to the config schema of the
returned resource definition. If not set, Dagster will accept any config provided for
the resource.
For example:
```python
@op(required_resource_keys={"globals"})
def my_op(context):
print(context.resources.globals["my_str_var"])
@job(resource_defs={"globals": make_values_resource(my_str_var=str, my_int_var=int)})
def my_job():
my_op()
```
Returns: A resource that passes in user-defined values.Return type: [ResourceDefinition](#dagster.ResourceDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_init_resource_context'>dagster.build_init_resource_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/init.py#L262' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_init_resource_context" class="hash-link"></a></Link></dt>
<dd>
Builds resource initialization context from provided parameters.
`build_init_resource_context` can be used as either a function or context manager. If there is a
provided resource to `build_init_resource_context` that is a context manager, then it must be
used as a context manager. This function can be used to provide the context argument to the
invocation of a resource.
Parameters:
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The resources to provide to the context. These can be either values or resource definitions.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – The resource config to provide to the context.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured for the context. Defaults to DagsterInstance.ephemeral().
Examples:
```python
context = build_init_resource_context()
resource_to_init(context)
with build_init_resource_context(
resources={"foo": context_manager_resource}
) as context:
resource_to_init(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_resources'>dagster.build_resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/build_resources.py#L43' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_resources" class="hash-link"></a></Link></dt>
<dd>
Context manager that yields resources using provided resource definitions and run config.
This API allows for using resources in an independent context. Resources will be initialized
with the provided run config, and optionally, dagster_run. The resulting resources will be
yielded on a dictionary keyed identically to that provided for <cite>resource_defs</cite>. Upon exiting the
context, resources will also be torn down safely.
Parameters:
- <strong>resources</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – Resource instances or definitions to build. All required resource dependencies to a given resource must be contained within this dictionary, or the resource build will fail.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured to instantiate resources on.
- <strong>resource_config</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict representing the config to be provided to each resource during initialization and teardown.
- <strong>dagster_run</strong> (<em>Optional</em><em>[</em><em>PipelineRun</em><em>]</em>) – The pipeline run to provide during resource initialization and teardown. If the provided resources require either the <cite>dagster_run</cite> or <cite>run_id</cite> attributes of the provided context during resource initialization and/or teardown, this must be provided, or initialization will fail.
- <strong>log_manager</strong> (<em>Optional</em><em>[</em>[*DagsterLogManager*](loggers.mdx#dagster.DagsterLogManager)<em>]</em>) – Log Manager to use during resource initialization. Defaults to system log manager.
- <strong>event_loop</strong> (<em>Optional</em><em>[</em><em>AbstractEventLoop</em><em>]</em>) – An event loop for handling resources with async context managers.
Examples:
```python
from dagster import resource, build_resources
@resource
def the_resource():
return "foo"
with build_resources(resources={"from_def": the_resource, "from_val": "bar"}) as resources:
assert resources.from_def == "foo"
assert resources.from_val == "bar"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.with_resources'>dagster.with_resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/with_resources.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.with_resources" class="hash-link"></a></Link></dt>
<dd>
Adds dagster resources to copies of resource-requiring dagster definitions.
An error will be thrown if any provided definitions have a conflicting
resource definition provided for a key provided to resource_defs. Resource
config can be provided, with keys in the config dictionary corresponding to
the keys for each resource definition. If any definition has unsatisfied
resource keys after applying with_resources, an error will be thrown.
Parameters:
- <strong>definitions</strong> (<em>Iterable</em><em>[</em><em>ResourceAddable</em><em>]</em>) – Dagster definitions to provide resources to.
- <strong>resource_defs</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em>) – Mapping of resource keys to objects to satisfy resource requirements of provided dagster definitions.
- <strong>resource_config_by_key</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Specifies config for provided resources. The key in this dictionary corresponds to configuring the same key in the resource_defs dictionary.
Examples:
```python
from dagster import asset, resource, with_resources
@resource(config_schema={"bar": str})
def foo_resource():
...
@asset(required_resource_keys={"foo"})
def asset1(context):
foo = context.resources.foo
...
@asset(required_resource_keys={"foo"})
def asset2(context):
foo = context.resources.foo
...
asset1_with_foo, asset2_with_foo = with_resources(
[asset1, asset2],
resource_defs={
"foo": foo_resource
},
resource_config_by_key={
"foo": {
"config": {"bar": ...}
}
}
)
```
</dd>
</dl>
</div>
<div class="section" id="utilities">
## Utilities
<dl>
<dt><Link class="anchor" id='dagster.EnvVar'>`class` dagster.EnvVar <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_config/field_utils.py#L525' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.EnvVar" class="hash-link"></a></Link></dt>
<dd>
Class used to represent an environment variable in the Dagster config system.
This class is intended to be used to populate config fields or resources.
The environment variable will be resolved to a string value when the config is
loaded.
To access the value of the environment variable, use the <cite>get_value</cite> method.
</dd>
</dl>
</div>
<div class="section" id="legacy-resource-system">
## Legacy resource system
The following classes are used as part of the [legacy resource system](https://legacy-docs.dagster.io/concepts/resources-legacy).
<dl>
<dt><Link class="anchor" id='dagster.resource'>@dagster.resource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/resource_definition.py#L371' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.resource" class="hash-link"></a></Link></dt>
<dd>
Define a resource.
The decorated function should accept an [`InitResourceContext`](#dagster.InitResourceContext) and return an instance of
the resource. This function will become the `resource_fn` of an underlying
[`ResourceDefinition`](#dagster.ResourceDefinition).
If the decorated function yields once rather than returning (in the manner of functions
decorable with `@contextlib.contextmanager`) then
the body of the function after the yield will be run after execution resolves, allowing users
to write their own teardown/cleanup logic.
Parameters:
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.resource_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the resource.
- <strong>version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) The version of a resource function. Two wrapped resource functions should only have the same version if they produce the same resource definition when provided with the same inputs.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Keys for the resources required by this resource.
</dd>
</dl>
</div></div>
---
---
title: 'schedules and sensors'
sidebar_position: 1000
title_meta: 'schedules and sensors API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'schedules and sensors Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="schedules-and-sensors">
# Schedules and sensors
Dagster offers several ways to run data pipelines without manual intervention, including traditional scheduling and event-based triggers. [Automating your Dagster pipelines](https://docs.dagster.io/guides/automate) can boost efficiency and ensure that data is produced consistently and reliably.
<div class="section" id="run-requests">
## Run requests
<dl>
<dt><Link class="anchor" id='dagster.RunRequest'>`class` dagster.RunRequest <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_request.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunRequest" class="hash-link"></a></Link></dt>
<dd>
Represents all the information required to launch a single run. Must be returned by a
SensorDefinition or ScheduleDefinition’s evaluation function for a run to be launched.
Parameters:
- <strong>run_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string key to identify this launched run. For sensors, ensures that only one run is created per run key across all sensor evaluations. For schedules, ensures that one run is created per tick, across failure recoveries. Passing in a <cite>None</cite> value means that a run will always be launched per evaluation.
- <strong>(</strong><strong>Optional</strong><strong>[</strong><strong>Union</strong><strong>[</strong><strong>RunConfig</strong> (<em>run_config</em>) – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>Mapping</strong><strong>[</strong><strong>str</strong> – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>Any</strong><strong>]</strong><strong>]</strong><strong>]</strong> – Configuration for the run. If the job has a [`PartitionedConfig`](partitions.mdx#dagster.PartitionedConfig), this value will override replace the config provided by it.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags (string key-value pairs) to attach to the launched run.
- <strong>job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the job this run request will launch. Required for sensors that target multiple jobs.
- <strong>asset_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – A subselection of assets that should be launched with this run. If the sensor or schedule targets a job, then by default a RunRequest returned from it will launch all of the assets in the job. If the sensor targets an asset selection, then by default a RunRequest returned from it will launch all the assets in the selection. This argument is used to specify that only a subset of these assets should be launched, instead of all of them.
- <strong>asset_check_keys</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*AssetCheckKey*](asset-checks.mdx#dagster.AssetCheckKey)<em>]</em><em>]</em>) – A subselection of asset checks that should be launched with this run. If the sensor/schedule targets a job, then by default a RunRequest returned from it will launch all of the asset checks in the job. If the sensor/schedule targets an asset selection, then by default a RunRequest returned from it will launch all the asset checks in the selection. This argument is used to specify that only a subset of these asset checks should be launched, instead of all of them.
- <strong>stale_assets_only</strong> (<em>bool</em>) – Set to true to further narrow the asset selection to stale assets. If passed without an asset selection, all stale assets in the job will be materialized. If the job does not materialize assets, this flag is ignored.
- <strong>partition_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The partition key for this run request.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SkipReason'>`class` dagster.SkipReason <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_request.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SkipReason" class="hash-link"></a></Link></dt>
<dd>
Represents a skipped evaluation, where no runs are requested. May contain a message to indicate
why no runs were requested.
Parameters: <strong>skip_message</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A message displayed in the Dagster UI for why this evaluation resulted
in no requested runs.
</dd>
</dl>
</div>
<div class="section" id="schedules">
## Schedules
[Schedules](https://docs.dagster.io/guides/automate/schedules) are Dagster’s way to support traditional ways of automation, such as specifying a job should run at Mondays at 9:00AM. Jobs triggered by schedules can contain a subset of [assets](https://docs.dagster.io/guides/build/assets) or [ops](https://legacy-docs.dagster.io/concepts/ops-jobs-graphs/ops).
<dl>
<dt><Link class="anchor" id='dagster.schedule'>@dagster.schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/schedule_decorator.py#L40' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.schedule" class="hash-link"></a></Link></dt>
<dd>
Creates a schedule following the provided cron schedule and requests runs for the provided job.
The decorated function takes in a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) as its only
argument, and does one of the following:
1. Return a [`RunRequest`](#dagster.RunRequest) object.
2. Return a list of [`RunRequest`](#dagster.RunRequest) objects.
3. Return a [`SkipReason`](#dagster.SkipReason) object, providing a descriptive message of why no runs were requested.
4. Return nothing (skipping without providing a reason)
5. Return a run config dictionary.
6. Yield a [`SkipReason`](#dagster.SkipReason) or yield one ore more [`RunRequest`](#dagster.RunRequest) objects.
Returns a [`ScheduleDefinition`](#dagster.ScheduleDefinition).
Parameters:
- <strong>cron_schedule</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A valid cron string or sequence of cron strings specifying when the schedule will run, e.g., `45 23 * * 6` for a schedule that runs at 11:45 PM every Saturday. If a sequence is provided, then the schedule will run for the union of all execution times for the provided cron strings, e.g., `['45 23 * * 6', '30 9 * * 0']` for a schedule that runs at 11:45 PM every Saturday and 9:30 AM every Sunday.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the schedule.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the schedule and can be used for searching and filtering in the UI.
- <strong>tags_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em><em>, </em><em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – A function that generates tags to attach to the schedule’s runs. Takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) and returns a dictionary of tags (string key-value pairs). <strong>Note</strong>: Either `tags` or `tags_fn` may be set, but not both.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the schedule. Values will be normalized to typed <cite>MetadataValue</cite> objects.
- <strong>should_execute</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that runs at schedule execution time to determine whether a schedule should execute or skip. Takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) and returns a boolean (`True` if the schedule should execute). Defaults to a function that always returns `True`.
- <strong>execution_timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Timezone in which the schedule should run. Supported strings for timezones are the ones provided by the [IANA time zone database](https://www.iana.org/time-zones) - e.g. `"America/Los_Angeles"`.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the schedule.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job that should execute when the schedule runs.
- <strong>default_status</strong> (<em>DefaultScheduleStatus</em>) – If set to `RUNNING`, the schedule will immediately be active when starting Dagster. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The set of resource keys required by the schedule.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetSelection</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The target that the schedule will execute. It can take [`AssetSelection`](assets.mdx#dagster.AssetSelection) objects and anything coercible to it (e.g. <cite>str</cite>, <cite>Sequence[str]</cite>, <cite>AssetKey</cite>, <cite>AssetsDefinition</cite>). It can also accept [`JobDefinition`](jobs.mdx#dagster.JobDefinition) (a function decorated with <cite>@job</cite> is an instance of <cite>JobDefinition</cite>) and <cite>UnresolvedAssetJobDefinition</cite> (the return value of [`define_asset_job()`](assets.mdx#dagster.define_asset_job)) objects. This parameter will replace <cite>job</cite> and <cite>job_name</cite>.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A list of strings representing owners of the schedule. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition'>`class` dagster.ScheduleDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L870' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition" class="hash-link"></a></Link></dt>
<dd>
Defines a schedule that targets a job.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the schedule to create. Defaults to the job name plus `_schedule`.
- <strong>cron_schedule</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A valid cron string or sequence of cron strings specifying when the schedule will run, e.g., `45 23 * * 6` for a schedule that runs at 11:45 PM every Saturday. If a sequence is provided, then the schedule will run for the union of all execution times for the provided cron strings, e.g., `['45 23 * * 6', '30 9 * * 0]` for a schedule that runs at 11:45 PM every Saturday and 9:30 AM every Sunday.
- <strong>execution_fn</strong> (<em>Callable</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em>) –
The core evaluation function for the schedule, which is run at an interval to determine whether a run should be launched or not. Takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext).
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*RunConfig*](config.mdx#dagster.RunConfig)<em>, </em><em>Mapping</em><em>]</em><em>]</em>) – The config that parameterizes this execution, as a dict.
- <strong>run_config_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em><em>, </em><em>[</em><em>Mapping</em><em>]</em><em>]</em><em>]</em>) – A function that takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) object and returns the run configuration that parameterizes this execution, as a dict. <strong>Note</strong>: Only one of the following may be set: You may set `run_config`, `run_config_fn`, or `execution_fn`.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the schedule and can be used for searching and filtering in the UI. If no <cite>execution_fn</cite> is provided, then these will also be automatically attached to runs launched by the schedule.
- <strong>tags_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em><em>, </em><em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – A function that generates tags to attach to the schedule’s runs. Takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) and returns a dictionary of tags (string key-value pairs). <strong>Note</strong>: Only one of the following may be set: `tags`, `tags_fn`, or `execution_fn`.
- <strong>should_execute</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*ScheduleEvaluationContext*](#dagster.ScheduleEvaluationContext)<em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that runs at schedule execution time to determine whether a schedule should execute or skip. Takes a [`ScheduleEvaluationContext`](#dagster.ScheduleEvaluationContext) and returns a boolean (`True` if the schedule should execute). Defaults to a function that always returns `True`.
- <strong>execution_timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the schedule.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>]</em><em>]</em>) – The job that should execute when this schedule runs.
- <strong>default_status</strong> (<em>DefaultScheduleStatus</em>) – If set to `RUNNING`, the schedule will start as running. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The set of resource keys required by the schedule.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetSelection</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The target that the schedule will execute. It can take [`AssetSelection`](assets.mdx#dagster.AssetSelection) objects and anything coercible to it (e.g. <cite>str</cite>, <cite>Sequence[str]</cite>, <cite>AssetKey</cite>, <cite>AssetsDefinition</cite>). It can also accept [`JobDefinition`](jobs.mdx#dagster.JobDefinition) (a function decorated with <cite>@job</cite> is an instance of <cite>JobDefinition</cite>) and <cite>UnresolvedAssetJobDefinition</cite> (the return value of [`define_asset_job()`](assets.mdx#dagster.define_asset_job)) objects. This parameter will replace <cite>job</cite> and <cite>job_name</cite>.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the schedule. Values will be normalized to typed <cite>MetadataValue</cite> objects. Not currently shown in the UI but available at runtime via <cite>ScheduleEvaluationContext.repository_def.get_schedule_def(\<name>).metadata</cite>.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A list of strings representing owners of the schedule. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.cron_schedule'>`property` cron_schedule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L914' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.cron_schedule" class="hash-link"></a></Link></dt>
<dd>
The cron schedule representing when this schedule will be evaluated.
Type: Union[str, Sequence[str]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.default_status'>`property` default_status <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L1063' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.default_status" class="hash-link"></a></Link></dt>
<dd>
The default status for this schedule when it is first loaded in
a code location.
Type: DefaultScheduleStatus
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L908' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.description" class="hash-link"></a></Link></dt>
<dd>
A description for this schedule.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.environment_vars'>`property` environment_vars <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/decorator_utils.py#L920' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.environment_vars" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
Setting this property no longer has any effect..
:::
Environment variables to export to the cron schedule.
Type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.execution_timezone'>`property` execution_timezone <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L936' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.execution_timezone" class="hash-link"></a></Link></dt>
<dd>
The timezone in which this schedule will be evaluated.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.job'>`property` job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L962' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.job" class="hash-link"></a></Link></dt>
<dd>
The job that is
targeted by this schedule.
Type: Union[[JobDefinition](jobs.mdx#dagster.JobDefinition), UnresolvedAssetJobDefinition]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L902' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the job targeted by this schedule.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.metadata'>`property` metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L948' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.metadata" class="hash-link"></a></Link></dt>
<dd>
The metadata for this schedule.
Type: Mapping[str, str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L896' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.name" class="hash-link"></a></Link></dt>
<dd>
The name of the schedule.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L930' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
The set of keys for resources that must be provided to this schedule.
Type: Set[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleDefinition.tags'>`property` tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L942' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleDefinition.tags" class="hash-link"></a></Link></dt>
<dd>
The tags for this schedule.
Type: Mapping[str, str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleEvaluationContext'>`class` dagster.ScheduleEvaluationContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L146' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleEvaluationContext" class="hash-link"></a></Link></dt>
<dd>
The context object available as the first argument to various functions defined on a [`dagster.ScheduleDefinition`](#dagster.ScheduleDefinition).
A `ScheduleEvaluationContext` object is passed as the first argument to `run_config_fn`, `tags_fn`,
and `should_execute`.
<strong>Users should not instantiate this object directly</strong>. To construct a `ScheduleEvaluationContext` for testing purposes, use [`dagster.build_schedule_context()`](#dagster.build_schedule_context).
Example:
```python
from dagster import schedule, ScheduleEvaluationContext
@schedule
def the_schedule(context: ScheduleEvaluationContext):
...
```
<dl>
<dt><Link class="anchor" id='dagster.ScheduleEvaluationContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L298' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleEvaluationContext.instance" class="hash-link"></a></Link></dt>
<dd>
The current [`DagsterInstance`](internals.mdx#dagster.DagsterInstance).
Type: [DagsterInstance](internals.mdx#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleEvaluationContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L237' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleEvaluationContext.resources" class="hash-link"></a></Link></dt>
<dd>
Mapping of resource key to resource definition to be made available
during schedule execution.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.ScheduleEvaluationContext.scheduled_execution_time'>`property` scheduled_execution_time <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L319' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.ScheduleEvaluationContext.scheduled_execution_time" class="hash-link"></a></Link></dt>
<dd>
The time in which the execution was scheduled to happen. May differ slightly
from both the actual execution time and the time at which the run config is computed.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_schedule_context'>dagster.build_schedule_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/schedule_definition.py#L382' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_schedule_context" class="hash-link"></a></Link></dt>
<dd>
Builds schedule execution context using the provided parameters.
The instance provided to `build_schedule_context` must be persistent;
[`DagsterInstance.ephemeral()`](internals.mdx#dagster.DagsterInstance) will result in an error.
Parameters:
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The Dagster instance configured to run the schedule.
- <strong>scheduled_execution_time</strong> (<em>datetime</em>) – The time in which the execution was scheduled to happen. May differ slightly from both the actual execution time and the time at which the run config is computed.
Examples:
```python
context = build_schedule_context(instance)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_schedule_from_partitioned_job'>dagster.build_schedule_from_partitioned_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/partitions/partitioned_schedule.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_schedule_from_partitioned_job" class="hash-link"></a></Link></dt>
<dd>
Creates a schedule from a job that targets
time window-partitioned or statically-partitioned assets. The job can also be
multi-partitioned, as long as one of the partition dimensions is time-partitioned.
The schedule executes at the cadence specified by the time partitioning of the job or assets.
<strong>Example:</strong>
```python
######################################
# Job that targets partitioned assets
######################################
from dagster import (
DailyPartitionsDefinition,
asset,
build_schedule_from_partitioned_job,
define_asset_job,
Definitions,
)
@asset(partitions_def=DailyPartitionsDefinition(start_date="2020-01-01"))
def asset1():
...
asset1_job = define_asset_job("asset1_job", selection=[asset1])
# The created schedule will fire daily
asset1_job_schedule = build_schedule_from_partitioned_job(asset1_job)
Definitions(assets=[asset1], schedules=[asset1_job_schedule])
################
# Non-asset job
################
from dagster import DailyPartitionsDefinition, build_schedule_from_partitioned_job, jog
@job(partitions_def=DailyPartitionsDefinition(start_date="2020-01-01"))
def do_stuff_partitioned():
...
# The created schedule will fire daily
do_stuff_partitioned_schedule = build_schedule_from_partitioned_job(
do_stuff_partitioned,
)
Definitions(schedules=[do_stuff_partitioned_schedule])
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._core.scheduler.DagsterDaemonScheduler'>dagster._core.scheduler.DagsterDaemonScheduler Scheduler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/scheduler/scheduler.py#L266' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._core.scheduler.DagsterDaemonScheduler" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Default scheduler implementation that submits runs from the long-lived `dagster-daemon`
process. Periodically checks each running schedule for execution times that don’t yet
have runs and launches them.
</dd>
</dl>
</div>
<div class="section" id="sensors">
## Sensors
[Sensors](https://docs.dagster.io/guides/automate/sensors) are typically used to poll, listen, and respond to external events. For example, you could configure a sensor to run a job or materialize an asset in response to specific events.
<dl>
<dt><Link class="anchor" id='dagster.sensor'>@dagster.sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/sensor_decorator.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.sensor" class="hash-link"></a></Link></dt>
<dd>
Creates a sensor where the decorated function is used as the sensor’s evaluation function.
The decorated function may:
1. Return a <cite>RunRequest</cite> object.
2. Return a list of <cite>RunRequest</cite> objects.
3. Return a <cite>SkipReason</cite> object, providing a descriptive message of why no runs were requested.
4. Return nothing (skipping without providing a reason)
5. Yield a <cite>SkipReason</cite> or yield one or more <cite>RunRequest</cite> objects.
Takes a `SensorEvaluationContext`.
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to the name of the decorated function.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job to be executed when the sensor fires.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed when the sensor fires.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>asset_selection</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em><em>]</em>) – An asset selection to launch a run for if the sensor condition is met. This can be provided instead of specifying a job.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A set of resource keys that must be available on the context when the sensor evaluation function runs. Use this to specify resources your sensor function depends on.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetSelection</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The target that the sensor will execute. It can take [`AssetSelection`](assets.mdx#dagster.AssetSelection) objects and anything coercible to it (e.g. <cite>str</cite>, <cite>Sequence[str]</cite>, <cite>AssetKey</cite>, <cite>AssetsDefinition</cite>). It can also accept [`JobDefinition`](jobs.mdx#dagster.JobDefinition) (a function decorated with <cite>@job</cite> is an instance of <cite>JobDefinition</cite>) and <cite>UnresolvedAssetJobDefinition</cite> (the return value of [`define_asset_job()`](assets.mdx#dagster.define_asset_job)) objects. This is a parameter that will replace <cite>job</cite>, <cite>jobs</cite>, and <cite>asset_selection</cite>.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A list of strings representing owners of the sensor. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition'>`class` dagster.SensorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L826' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition" class="hash-link"></a></Link></dt>
<dd>
Define a sensor that initiates a set of runs based on some external state.
Parameters:
- <strong>evaluation_fn</strong> (<em>Callable</em><em>[</em><em>[</em><em>SensorEvaluationContext</em><em>]</em><em>]</em>) –
The core evaluation function for the sensor, which is run at an interval to determine whether a run should be launched or not. Takes a `SensorEvaluationContext`.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor to create. Defaults to name of evaluation_fn
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJob</em><em>]</em>) – The job to execute when this sensor fires.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJob</em><em>]</em><em>]</em>) – A list of jobs to execute when this sensor fires.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>asset_selection</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](assets.mdx#dagster.SourceAsset)<em>]</em><em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em><em>]</em>) – An asset selection to launch a run for if the sensor condition is met. This can be provided instead of specifying a job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects. Not currently shown in the UI but available at runtime via <cite>SensorEvaluationContext.repository_def.get_sensor_def(\<name>).metadata</cite>.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>CoercibleToAssetSelection</em><em>, </em>[*AssetsDefinition*](assets.mdx#dagster.AssetsDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The target that the sensor will execute. It can take [`AssetSelection`](assets.mdx#dagster.AssetSelection) objects and anything coercible to it (e.g. <cite>str</cite>, <cite>Sequence[str]</cite>, <cite>AssetKey</cite>, <cite>AssetsDefinition</cite>). It can also accept [`JobDefinition`](jobs.mdx#dagster.JobDefinition) (a function decorated with <cite>@job</cite> is an instance of <cite>JobDefinition</cite>) and <cite>UnresolvedAssetJobDefinition</cite> (the return value of [`define_asset_job()`](assets.mdx#dagster.define_asset_job)) objects. This is a parameter that will replace <cite>job</cite>, <cite>jobs</cite>, and <cite>asset_selection</cite>.
- <strong>owners</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A list of strings representing owners of the sensor. Each string can be a user’s email address, or a team name prefixed with <cite>team:</cite>, e.g. <cite>team:finops</cite>.
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.default_status'>`property` default_status <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L1174' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.default_status" class="hash-link"></a></Link></dt>
<dd>
The default status for this sensor when it is first loaded in
a code location.
Type: DefaultSensorStatus
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L851' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.description" class="hash-link"></a></Link></dt>
<dd>
A description for this sensor.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.job'>`property` job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L867' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.job" class="hash-link"></a></Link></dt>
<dd>
The job that is
targeted by this schedule.
Type: Union[[GraphDefinition](graphs.mdx#dagster.GraphDefinition), [JobDefinition](jobs.mdx#dagster.JobDefinition), UnresolvedAssetJobDefinition]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L1162' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the job that is targeted by this sensor.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.jobs'>`property` jobs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L892' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.jobs" class="hash-link"></a></Link></dt>
<dd>
A list of jobs
that are targeted by this schedule.
Type: List[Union[[GraphDefinition](graphs.mdx#dagster.GraphDefinition), [JobDefinition](jobs.mdx#dagster.JobDefinition), UnresolvedAssetJobDefinition]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.minimum_interval_seconds'>`property` minimum_interval_seconds <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L857' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.minimum_interval_seconds" class="hash-link"></a></Link></dt>
<dd>
The minimum number of seconds between sequential evaluations of this sensor.
Type: Optional[int]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.name'>`property` name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L845' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.name" class="hash-link"></a></Link></dt>
<dd>
The name of this sensor.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorDefinition.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L839' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorDefinition.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
The set of keys for resources that must be provided to this sensor.
Type: Set[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt>`class` dagster.SensorEvaluationContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L103' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The context object available as the argument to the evaluation function of a [`dagster.SensorDefinition`](#dagster.SensorDefinition).
Users should not instantiate this object directly. To construct a
<cite>SensorEvaluationContext</cite> for testing purposes, use `dagster.
build_sensor_context()`.
Parameters:
- <strong>instance_ref</strong> (<em>Optional</em><em>[</em>[*InstanceRef*](internals.mdx#dagster._core.instance.InstanceRef)<em>]</em>) – The serialized instance configured to run the schedule
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The cursor, passed back from the last sensor evaluation via the cursor attribute of SkipReason and RunRequest
- <strong>last_tick_completion_time</strong> (<em>float</em>) – The last time that the sensor was evaluated (UTC).
- <strong>last_run_key</strong> (<em>str</em>) – DEPRECATED The run key of the RunRequest most recently created by this sensor. Use the preferred <cite>cursor</cite> attribute instead.
- <strong>log_key</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The log key to use for this sensor tick.
- <strong>repository_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the repository that the sensor belongs to.
- <strong>repository_def</strong> (<em>Optional</em><em>[</em>[*RepositoryDefinition*](repositories.mdx#dagster.RepositoryDefinition)<em>]</em>) – The repository or that the sensor belongs to. If needed by the sensor top-level resource definitions will be pulled from this repository. You can provide either this or <cite>definitions</cite>.
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The deserialized instance can also be passed in directly (primarily useful in testing contexts).
- <strong>definitions</strong> (<em>Optional</em><em>[</em>[*Definitions*](definitions.mdx#dagster.Definitions)<em>]</em>) – <cite>Definitions</cite> object that the sensor is defined in. If needed by the sensor, top-level resource definitions will be pulled from these definitions. You can provide either this or <cite>repository_def</cite>.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of resource keys to resource definitions to be made available during sensor execution.
- <strong>last_sensor_start_time</strong> (<em>float</em>) – The last time that the sensor was started (UTC).
- <strong>code_location_origin</strong> (<em>Optional</em><em>[</em><em>CodeLocationOrigin</em><em>]</em>) – The code location that the sensor is in.
Example:
```python
from dagster import sensor, SensorEvaluationContext
@sensor
def the_sensor(context: SensorEvaluationContext):
...
```
<dl>
<dt>update_cursor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L382' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Updates the cursor value for this sensor, which will be provided on the context for the
next sensor evaluation.
This can be used to keep track of progress and avoid duplicate work across sensor
evaluations.
Parameters: <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>)
</dd>
</dl>
<dl>
<dt>`property` cursor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L376' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The cursor value for this sensor, which was set in an earlier sensor evaluation.
</dd>
</dl>
<dl>
<dt>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L317' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The current DagsterInstance.
Type: [DagsterInstance](internals.mdx#dagster.DagsterInstance)
</dd>
</dl>
<dl>
<dt>`property` is_first_tick_since_sensor_start <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L361' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Flag representing if this is the first tick since the sensor was started.
</dd>
</dl>
<dl>
<dt>`property` last_run_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L370' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The run key supplied to the most recent RunRequest produced by this sensor.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt>`property` last_sensor_start_time <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L352' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Timestamp representing the last time this sensor was started. Can be
used in concert with last_tick_completion_time to determine if this is the first tick since the
sensor was started.
Type: Optional[float]
</dd>
</dl>
<dl>
<dt>`property` last_tick_completion_time <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L338' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
Timestamp representing the last time this sensor completed an evaluation.
Type: Optional[float]
</dd>
</dl>
<dl>
<dt>`property` repository_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L406' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The RepositoryDefinition that this sensor resides in.
Type: Optional[[RepositoryDefinition](repositories.mdx#dagster.RepositoryDefinition)]
</dd>
</dl>
<dl>
<dt>`property` repository_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L400' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
The name of the repository that this sensor resides in.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L266' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a></dt>
<dd>
A mapping from resource key to instantiated resources for this sensor.
Type: Resources
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_sensor_context'>dagster.build_sensor_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L1288' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_sensor_context" class="hash-link"></a></Link></dt>
<dd>
Builds sensor execution context using the provided parameters.
This function can be used to provide a context to the invocation of a sensor definition.If
provided, the dagster instance must be persistent; DagsterInstance.ephemeral() will result in an
error.
Parameters:
- <strong>instance</strong> (<em>Optional</em><em>[</em>[*DagsterInstance*](internals.mdx#dagster.DagsterInstance)<em>]</em>) – The dagster instance configured to run the sensor.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A cursor value to provide to the evaluation of the sensor.
- <strong>repository_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the repository that the sensor belongs to.
- <strong>repository_def</strong> (<em>Optional</em><em>[</em>[*RepositoryDefinition*](repositories.mdx#dagster.RepositoryDefinition)<em>]</em>) – The repository that the sensor belongs to. If needed by the sensor top-level resource definitions will be pulled from this repository. You can provide either this or <cite>definitions</cite>.
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – A set of resource definitions to provide to the sensor. If passed, these will override any resource definitions provided by the repository.
- <strong>definitions</strong> (<em>Optional</em><em>[</em>[*Definitions*](definitions.mdx#dagster.Definitions)<em>]</em>) – <cite>Definitions</cite> object that the sensor is defined in. If needed by the sensor, top-level resource definitions will be pulled from these definitions. You can provide either this or <cite>repository_def</cite>.
- <strong>last_sensor_start_time</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – The last time the sensor was started.
Examples:
```python
context = build_sensor_context()
my_sensor(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.asset_sensor'>@dagster.asset_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/sensor_decorator.py#L131' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.asset_sensor" class="hash-link"></a></Link></dt>
<dd>
Creates an asset sensor where the decorated function is used as the asset sensor’s evaluation
function.
If the asset has been materialized multiple times between since the last sensor tick, the
evaluation function will only be invoked once, with the latest materialization.
The decorated function may:
1. Return a <cite>RunRequest</cite> object.
2. Return a list of <cite>RunRequest</cite> objects.
3. Return a <cite>SkipReason</cite> object, providing a descriptive message of why no runs were requested.
4. Return nothing (skipping without providing a reason)
5. Yield a <cite>SkipReason</cite> or yield one or more <cite>RunRequest</cite> objects.
Takes a `SensorEvaluationContext` and an EventLogEntry corresponding to an
AssetMaterialization event.
Parameters:
- <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – The asset_key this sensor monitors.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to the name of the decorated function.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job to be executed when the sensor fires.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed when the sensor fires.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI. Values that are not already strings will be serialized as JSON.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
Example:
```python
from dagster import AssetKey, EventLogEntry, SensorEvaluationContext, asset_sensor
@asset_sensor(asset_key=AssetKey("my_table"), job=my_job)
def my_asset_sensor(context: SensorEvaluationContext, asset_event: EventLogEntry):
return RunRequest(
run_key=context.cursor,
run_config={
"ops": {
"read_materialization": {
"config": {
"asset_key": asset_event.dagster_event.asset_key.path,
}
}
}
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.multi_asset_sensor'>@dagster.multi_asset_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/decorators/sensor_decorator.py#L260' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.multi_asset_sensor" class="hash-link"></a></Link></dt>
<dd>
Creates an asset sensor that can monitor multiple assets.
The decorated function is used as the asset sensor’s evaluation
function. The decorated function may:
1. Return a <cite>RunRequest</cite> object.
2. Return a list of <cite>RunRequest</cite> objects.
3. Return a <cite>SkipReason</cite> object, providing a descriptive message of why no runs were requested.
4. Return nothing (skipping without providing a reason)
5. Yield a <cite>SkipReason</cite> or yield one or more <cite>RunRequest</cite> objects.
Takes a `MultiAssetSensorEvaluationContext`.
Parameters:
- <strong>monitored_assets</strong> (<em>Union</em><em>[</em><em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em><em>, </em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em>) – The assets this sensor monitors. If an AssetSelection object is provided, it will only apply to assets within the Definitions that this sensor is part of.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to the name of the decorated function.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job to be executed when the sensor fires.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed when the sensor fires.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>request_assets</strong> (<em>Optional</em><em>[</em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em>) – An asset selection to launch a run for if the sensor condition is met. This can be provided instead of specifying a job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.run_status_sensor'>@dagster.run_status_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L1089' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.run_status_sensor" class="hash-link"></a></Link></dt>
<dd>
Creates a sensor that reacts to a given status of job execution, where the decorated
function will be run when a job is at the given status.
Takes a [`RunStatusSensorContext`](#dagster.RunStatusSensorContext).
Parameters:
- <strong>run_status</strong> ([*DagsterRunStatus*](internals.mdx#dagster.DagsterRunStatus)) – The status of run execution which will be monitored by the sensor.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to the name of the decorated function.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>, </em>[*RepositorySelector*](#dagster.RepositorySelector)<em>, </em>[*JobSelector*](#dagster.JobSelector)<em>, </em><em>CodeLocationSelector</em><em>]</em><em>]</em><em>]</em>) – Jobs in the current code locations that will be monitored by this sensor. Defaults to None, which means the alert will be sent when any job in the code location matches the requested run_status. Jobs in external repositories can be monitored by using RepositorySelector or JobSelector.
- <strong>monitor_all_code_locations</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>job_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*RepositorySelector*](#dagster.RepositorySelector)<em>, </em>[*JobSelector*](#dagster.JobSelector)<em>, </em><em>CodeLocationSelector</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitored_jobs) Jobs in the current code location that will be monitored by this sensor. Defaults to None, which means the alert will be sent when any job in the code location matches the requested run_status.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>request_job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job that should be executed if a RunRequest is yielded from the sensor.
- <strong>request_jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed if RunRequests are yielded from the sensor.
- <strong>monitor_all_repositories</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitor_all_code_locations) If set to True, the sensor will monitor all runs in the Dagster instance. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.run_failure_sensor'>@dagster.run_failure_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L439' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.run_failure_sensor" class="hash-link"></a></Link></dt>
<dd>
Creates a sensor that reacts to job failure events, where the decorated function will be
run when a run fails.
Takes a [`RunFailureSensorContext`](#dagster.RunFailureSensorContext).
Parameters:
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the job failure sensor. Defaults to the name of the decorated function.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>, </em>[*RepositorySelector*](#dagster.RepositorySelector)<em>, </em>[*JobSelector*](#dagster.JobSelector)<em>, </em><em>CodeLocationSelector</em><em>]</em><em>]</em><em>]</em>) – The jobs in the current repository that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the current repository fails.
- <strong>monitor_all_code_locations</strong> (<em>bool</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>job_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*RepositorySelector*](#dagster.RepositorySelector)<em>, </em>[*JobSelector*](#dagster.JobSelector)<em>, </em><em>CodeLocationSelector</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitored_jobs) The jobs in the current repository that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the repository fails.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>request_job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJob</em><em>]</em><em>]</em>) – The job a RunRequest should execute if yielded from the sensor.
- <strong>request_jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJob</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed if RunRequests are yielded from the sensor.
- <strong>monitor_all_repositories</strong> (<em>bool</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitor_all_code_locations) If set to True, the sensor will monitor all runs in the Dagster instance. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AssetSensorDefinition'>`class` dagster.AssetSensorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/sensor_definition.py#L826' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSensorDefinition" class="hash-link"></a></Link></dt>
<dd>
Define an asset sensor that initiates a set of runs based on the materialization of a given
asset.
If the asset has been materialized multiple times between since the last sensor tick, the
evaluation function will only be invoked once, with the latest materialization.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the sensor to create.
- <strong>asset_key</strong> ([*AssetKey*](assets.mdx#dagster.AssetKey)) – The asset_key this sensor monitors.
- <strong>asset_materialization_fn</strong> (<em>Callable</em><em>[</em><em>[</em><em>SensorEvaluationContext</em><em>, </em>[*EventLogEntry*](internals.mdx#dagster.EventLogEntry)<em>]</em><em>, </em><em>Union</em><em>[</em><em>Iterator</em><em>[</em><em>Union</em><em>[</em>[*RunRequest*](#dagster.RunRequest)<em>, </em>[*SkipReason*](#dagster.SkipReason)<em>]</em><em>]</em><em>, </em>[*RunRequest*](#dagster.RunRequest)<em>, </em>[*SkipReason*](#dagster.SkipReason)<em>]</em><em>]</em>) –
The core evaluation function for the sensor, which is run at an interval to determine whether a run should be launched or not. Takes a `SensorEvaluationContext` and an EventLogEntry corresponding to an AssetMaterialization event.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job object to target with this sensor.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed when the sensor fires.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
<dl>
<dt><Link class="anchor" id='dagster.AssetSensorDefinition.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/asset_sensor_definition.py#L183' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AssetSensorDefinition.asset_key" class="hash-link"></a></Link></dt>
<dd>
The key of the asset targeted by this sensor.
Type: [AssetKey](assets.mdx#dagster.AssetKey)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.MultiAssetSensorDefinition'>`class` dagster.MultiAssetSensorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/multi_asset_sensor_definition.py#L1269' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.MultiAssetSensorDefinition" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
For most use cases, Declarative Automation should be used instead of multi_asset_sensors to monitor the status of upstream assets and launch runs in response. In cases where side effects are required, or a specific job must be targeted for execution, multi_asset_sensors may be used..
:::
Define an asset sensor that initiates a set of runs based on the materialization of a list of
assets.
Users should not instantiate this object directly. To construct a
<cite>MultiAssetSensorDefinition</cite>, use `dagster.
multi_asset_sensor()`.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the sensor to create.
- <strong>asset_keys</strong> (<em>Sequence</em><em>[</em>[*AssetKey*](assets.mdx#dagster.AssetKey)<em>]</em>) – The asset_keys this sensor monitors.
- <strong>asset_materialization_fn</strong> (<em>Callable</em><em>[</em><em>[</em><em>MultiAssetSensorEvaluationContext</em><em>]</em><em>, </em><em>Union</em><em>[</em><em>Iterator</em><em>[</em><em>Union</em><em>[</em>[*RunRequest*](#dagster.RunRequest)<em>, </em>[*SkipReason*](#dagster.SkipReason)<em>]</em><em>]</em><em>, </em>[*RunRequest*](#dagster.RunRequest)<em>, </em>[*SkipReason*](#dagster.SkipReason)<em>]</em><em>]</em>) –
The core evaluation function for the sensor, which is run at an interval to determine whether a run should be launched or not. Takes a `MultiAssetSensorEvaluationContext`.
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em>) – The job object to target with this sensor.
- <strong>jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed when the sensor fires.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>request_assets</strong> (<em>Optional</em><em>[</em>[*AssetSelection*](assets.mdx#dagster.AssetSelection)<em>]</em>) – an asset selection to launch a run for if the sensor condition is met. This can be provided instead of specifying a job.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorDefinition'>`class` dagster.RunStatusSensorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L996' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorDefinition" class="hash-link"></a></Link></dt>
<dd>
Define a sensor that reacts to a given status of job execution, where the decorated
function will be evaluated when a run is at the given status.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the sensor. Defaults to the name of the decorated function.
- <strong>run_status</strong> ([*DagsterRunStatus*](internals.mdx#dagster.DagsterRunStatus)) – The status of a run which will be monitored by the sensor.
- <strong>run_status_sensor_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*RunStatusSensorContext*](#dagster.RunStatusSensorContext)<em>]</em><em>, </em><em>Union</em><em>[</em>[*SkipReason*](#dagster.SkipReason)<em>, </em><em>DagsterRunReaction</em><em>]</em><em>]</em>) – The core evaluation function for the sensor. Takes a [`RunStatusSensorContext`](#dagster.RunStatusSensorContext).
- <strong>minimum_interval_seconds</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The minimum number of seconds that will elapse between sensor evaluations.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of the sensor.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>, </em>[*JobSelector*](#dagster.JobSelector)<em>, </em>[*RepositorySelector*](#dagster.RepositorySelector)<em>, </em><em>CodeLocationSelector</em><em>]</em><em>]</em><em>]</em>) – The jobs in the current repository that will be monitored by this sensor. Defaults to None, which means the alert will be sent when any job in the repository fails.
- <strong>monitor_all_code_locations</strong> (<em>bool</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>request_job</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>]</em><em>]</em>) – The job a RunRequest should execute if yielded from the sensor.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A set of key-value tags that annotate the sensor and can be used for searching and filtering in the UI.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A set of metadata entries that annotate the sensor. Values will be normalized to typed <cite>MetadataValue</cite> objects.
- <strong>request_jobs</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>]</em><em>]</em><em>]</em>) – A list of jobs to be executed if RunRequests are yielded from the sensor.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext'>`class` dagster.RunStatusSensorContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L124' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object available to a decorated function of `run_status_sensor`.
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.dagster_event'>`property` dagster_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L232' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.dagster_event" class="hash-link"></a></Link></dt>
<dd>
The event associated with the job run status.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.dagster_run'>`property` dagster_run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L226' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.dagster_run" class="hash-link"></a></Link></dt>
<dd>
The run of the job.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.instance'>`property` instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L238' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.instance" class="hash-link"></a></Link></dt>
<dd>
The current instance.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L244' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.log" class="hash-link"></a></Link></dt>
<dd>
The logger for the current sensor evaluation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L253' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key of the relevant run.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunStatusSensorContext.sensor_name'>`property` sensor_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunStatusSensorContext.sensor_name" class="hash-link"></a></Link></dt>
<dd>
The name of the sensor.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunFailureSensorContext'>`class` dagster.RunFailureSensorContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L296' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunFailureSensorContext" class="hash-link"></a></Link></dt>
<dd>
The `context` object available to a decorated function of `run_failure_sensor`.
Parameters:
- <strong>sensor_name</strong> (<em>str</em>) – the name of the sensor.
- <strong>dagster_run</strong> ([*DagsterRun*](internals.mdx#dagster.DagsterRun)) – the failed run.
<dl>
<dt><Link class="anchor" id='dagster.RunFailureSensorContext.get_step_failure_events'>get_step_failure_events <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L315' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunFailureSensorContext.get_step_failure_events" class="hash-link"></a></Link></dt>
<dd>
The step failure event for each step in the run that failed.
Examples:
```python
error_strings_by_step_key = {
# includes the stack trace
event.step_key: event.event_specific_data.error.to_string()
for event in context.get_step_failure_events()
}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RunFailureSensorContext.failure_event'>`property` failure_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L305' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RunFailureSensorContext.failure_event" class="hash-link"></a></Link></dt>
<dd>
The run failure event.
If the run failed because of an error inside a step, get_step_failure_events will have more
details on the step failure.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.JobSelector'>`class` dagster.JobSelector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/selector.py#L95' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.JobSelector" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.RepositorySelector'>`class` dagster.RepositorySelector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/selector.py#L152' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.RepositorySelector" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.build_run_status_sensor_context'>dagster.build_run_status_sensor_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_status_sensor_definition.py#L334' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.build_run_status_sensor_context" class="hash-link"></a></Link></dt>
<dd>
Builds run status sensor context from provided parameters.
This function can be used to provide the context argument when directly invoking a function
decorated with <cite>@run_status_sensor</cite> or <cite>@run_failure_sensor</cite>, such as when writing unit tests.
Parameters:
- <strong>sensor_name</strong> (<em>str</em>) – The name of the sensor the context is being constructed for.
- <strong>dagster_event</strong> ([*DagsterEvent*](execution.mdx#dagster.DagsterEvent)) – A DagsterEvent with the same event type as the one that triggers the run_status_sensor
- <strong>dagster_instance</strong> ([*DagsterInstance*](internals.mdx#dagster.DagsterInstance)) – The dagster instance configured for the context.
- <strong>dagster_run</strong> ([*DagsterRun*](internals.mdx#dagster.DagsterRun)) – DagsterRun object from running a job
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>object</em><em>]</em><em>]</em>) – A dictionary of resources to be made available to the sensor.
- <strong>repository_def</strong> (<em>Optional</em><em>[</em>[*RepositoryDefinition*](repositories.mdx#dagster.RepositoryDefinition)<em>]</em>) – <span className="flag flag-info">beta</span> The repository that the sensor belongs to.
Examples:
```python
instance = DagsterInstance.ephemeral()
result = my_job.execute_in_process(instance=instance)
dagster_run = result.dagster_run
dagster_event = result.get_job_success_event() # or get_job_failure_event()
context = build_run_status_sensor_context(
sensor_name="run_status_sensor_to_invoke",
dagster_instance=instance,
dagster_run=dagster_run,
dagster_event=dagster_event,
)
run_status_sensor_to_invoke(context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SensorResult'>`class` dagster.SensorResult <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/run_request.py#L298' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SensorResult" class="hash-link"></a></Link></dt>
<dd>
The result of a sensor evaluation.
Parameters:
- <strong>run_requests</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*RunRequest*](#dagster.RunRequest)<em>]</em><em>]</em>) – A list of run requests to be executed.
- <strong>skip_reason</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em>[*SkipReason*](#dagster.SkipReason)<em>]</em><em>]</em>) – A skip message indicating why sensor evaluation was skipped.
- <strong>cursor</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The cursor value for this sensor, which will be provided on the context for the next sensor evaluation.
- <strong>dynamic_partitions_requests</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*DeleteDynamicPartitionsRequest*](#dagster.DeleteDynamicPartitionsRequest)<em>, </em>[*AddDynamicPartitionsRequest*](#dagster.AddDynamicPartitionsRequest)<em>]</em><em>]</em><em>]</em>) – A list of dynamic partition requests to request dynamic partition addition and deletion. Run requests will be evaluated using the state of the partitions with these changes applied. We recommend limiting partition additions and deletions to a maximum of 25K partitions per sensor evaluation, as this is the maximum recommended partition limit per asset.
- <strong>asset_events</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetObservation*](assets.mdx#dagster.AssetObservation)<em>, </em>[*AssetMaterialization*](ops.mdx#dagster.AssetMaterialization)<em>, </em><em>AssetCheckEvaluation</em><em>]</em><em>]</em><em>]</em>) – A list of materializations, observations, and asset check evaluations that the system will persist on your behalf at the end of sensor evaluation. These events will be not be associated with any particular run, but will be queryable and viewable in the asset catalog.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.AddDynamicPartitionsRequest'>`class` dagster.AddDynamicPartitionsRequest <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dynamic_partitions_request.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.AddDynamicPartitionsRequest" class="hash-link"></a></Link></dt>
<dd>
A request to add partitions to a dynamic partitions definition, to be evaluated by a sensor or schedule.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DeleteDynamicPartitionsRequest'>`class` dagster.DeleteDynamicPartitionsRequest <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/dynamic_partitions_request.py#L35' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DeleteDynamicPartitionsRequest" class="hash-link"></a></Link></dt>
<dd>
A request to delete partitions to a dynamic partitions definition, to be evaluated by a sensor or schedule.
</dd>
</dl>
</div></div>
---
---
title: 'types'
sidebar_position: 1000
title_meta: 'types API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'types Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="types">
# Types
Dagster includes facilities for typing the input and output values of ops (“runtime” types).
<div class="section" id="built-in-types">
<Link id="builtin"></Link>
## Built-in types
<dl>
<dt><Link class="anchor" id='dagster.Nothing'>dagster.Nothing<a href="#dagster.Nothing" class="hash-link"></a></Link></dt>
<dd>
Use this type only for inputs and outputs, in order to establish an execution dependency without
communicating a value. Inputs of this type will not be passed to the op compute function, so
it is necessary to use the explicit [`In`](ops.mdx#dagster.In) API to define them rather than
the Python 3 type hint syntax.
All values are considered to be instances of `Nothing`.
<strong>Examples:</strong>
```python
@op
def wait(_) -> Nothing:
time.sleep(1)
return
@op(
ins={"ready": In(dagster_type=Nothing)},
)
def done(_) -> str:
return 'done'
@job
def nothing_job():
done(wait())
# Any value will pass the type check for Nothing
@op
def wait_int(_) -> Int:
time.sleep(1)
return 1
@job
def nothing_int_job():
done(wait_int())
```
</dd>
</dl>
</div>
<div class="section" id="making-new-types">
## Making new types
<dl>
<dt><Link class="anchor" id='dagster.DagsterType'>`class` dagster.DagsterType <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType" class="hash-link"></a></Link></dt>
<dd>
Define a type in dagster. These can be used in the inputs and outputs of ops.
Parameters:
- <strong>type_check_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*TypeCheckContext*](execution.mdx#dagster.TypeCheckContext)<em>, </em><em>Any</em><em>]</em><em>, </em><em>[</em><em>Union</em><em>[</em><em>bool</em><em>, </em>[*TypeCheck*](ops.mdx#dagster.TypeCheck)<em>]</em><em>]</em><em>]</em>) – The function that defines the type check. It takes the value flowing through the input or output of the op. If it passes, return either `True` or a [`TypeCheck`](ops.mdx#dagster.TypeCheck) with `success` set to `True`. If it fails, return either `False` or a [`TypeCheck`](ops.mdx#dagster.TypeCheck) with `success` set to `False`. The first argument must be named `context` (or, if unused, `_`, `_context`, or `context_`). Use `required_resource_keys` for access to resources.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
The unique key to identify types programmatically. The key property always has a value. If you omit key to the argument to the init function, it instead receives the value of `name`. If neither `key` nor `name` is provided, a `CheckError` is thrown.
In the case of a generic type such as `List` or `Optional`, this is generated programmatically based on the type parameters.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A unique name given by a user. If `key` is `None`, `key` becomes this value. Name is not given in a case where the user does not specify a unique name for this type, such as a generic class.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A markdown-formatted string, displayed in tooling.
- <strong>loader</strong> (<em>Optional</em><em>[</em>[*DagsterTypeLoader*](#dagster.DagsterTypeLoader)<em>]</em>) – An instance of a class that inherits from [`DagsterTypeLoader`](#dagster.DagsterTypeLoader) and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the [`@dagster_type_loader`](#dagster.dagster_type_loader) decorator to construct these arguments.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Resource keys required by the `type_check_fn`.
- <strong>is_builtin</strong> (<em>bool</em>) – Defaults to False. This is used by tools to display or filter built-in types (such as `String`, `Int`) to visually distinguish them from user-defined types. Meant for internal use.
- <strong>kind</strong> (<em>DagsterTypeKind</em>) – Defaults to None. This is used to determine the kind of runtime type for InputDefinition and OutputDefinition type checking.
- <strong>typing_type</strong> – Defaults to None. A valid python typing type (e.g. Optional[List[int]]) for the value contained within the DagsterType. Meant for internal use.
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.type_check'>type_check <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L166' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.type_check" class="hash-link"></a></Link></dt>
<dd>
Type check the value against the type.
Parameters:
- <strong>context</strong> ([*TypeCheckContext*](execution.mdx#dagster.TypeCheckContext)) – The context of the type check.
- <strong>value</strong> (<em>Any</em>) – The value to check.
Returns: The result of the type check.Return type: [TypeCheck](ops.mdx#dagster.TypeCheck)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.description'>`property` description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L249' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.description" class="hash-link"></a></Link></dt>
<dd>
Description of the type, or None if not provided.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.display_name'>`property` display_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L214' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.display_name" class="hash-link"></a></Link></dt>
<dd>
Either the name or key (if name is <cite>None</cite>) of the type, overridden in many subclasses.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.has_unique_name'>`property` has_unique_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.has_unique_name" class="hash-link"></a></Link></dt>
<dd>
Whether the type has a unique name.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.loader'>`property` loader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L243' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.loader" class="hash-link"></a></Link></dt>
<dd>
Loader for this type, if any.
Type: Optional[[DagsterTypeLoader](#dagster.DagsterTypeLoader)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.required_resource_keys'>`property` required_resource_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L208' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.required_resource_keys" class="hash-link"></a></Link></dt>
<dd>
Set of resource keys required by the type check function.
Type: AbstractSet[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.typing_type'>`property` typing_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L237' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.typing_type" class="hash-link"></a></Link></dt>
<dd>
The python typing type for this type.
Type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterType.unique_name'>`property` unique_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterType.unique_name" class="hash-link"></a></Link></dt>
<dd>
The unique name of this type. Can be None if the type is not unique, such as container types.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PythonObjectDagsterType'>dagster.PythonObjectDagsterType <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L524' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PythonObjectDagsterType" class="hash-link"></a></Link></dt>
<dd>
Define a type in dagster whose typecheck is an isinstance check.
Specifically, the type can either be a single python type (e.g. int),
or a tuple of types (e.g. (int, float)) which is treated as a union.
Examples:
```python
ntype = PythonObjectDagsterType(python_type=int)
assert ntype.name == 'int'
assert_success(ntype, 1)
assert_failure(ntype, 'a')
```
```python
ntype = PythonObjectDagsterType(python_type=(int, float))
assert ntype.name == 'Union[int, float]'
assert_success(ntype, 1)
assert_success(ntype, 1.5)
assert_failure(ntype, 'a')
```
Parameters:
- <strong>python_type</strong> (<em>Union</em><em>[</em><em>Type</em><em>, </em><em>Tuple</em><em>[</em><em>Type</em><em>, </em><em>...</em><em>]</em>) – The dagster typecheck function calls instanceof on this type.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name the type. Defaults to the name of `python_type`.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Key of the type. Defaults to name.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A markdown-formatted string, displayed in tooling.
- <strong>loader</strong> (<em>Optional</em><em>[</em>[*DagsterTypeLoader*](#dagster.DagsterTypeLoader)<em>]</em>) – An instance of a class that inherits from [`DagsterTypeLoader`](#dagster.DagsterTypeLoader) and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the [`@dagster_type_loader`](#dagster.dagster_type_loader) decorator to construct these arguments.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.dagster_type_loader'>dagster.dagster_type_loader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/config_schema.py#L84' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.dagster_type_loader" class="hash-link"></a></Link></dt>
<dd>
Create an dagster type loader that maps config data to a runtime value.
The decorated function should take the execution context and parsed config value and return the
appropriate runtime value.
Parameters: <strong>config_schema</strong> ([*ConfigSchema*](config.mdx#dagster.ConfigSchema)) – The schema for the config that’s passed to the decorated
function.
Examples:
```python
@dagster_type_loader(Permissive())
def load_dict(_context, value):
return value
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeLoader'>`class` dagster.DagsterTypeLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/config_schema.py#L20' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeLoader" class="hash-link"></a></Link></dt>
<dd>
Dagster type loaders are used to load unconnected inputs of the dagster type they are attached
to.
The recommended way to define a type loader is with the
[`@dagster_type_loader`](#dagster.dagster_type_loader) decorator.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeLoaderContext'>`class` dagster.DagsterTypeLoaderContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1361' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeLoaderContext" class="hash-link"></a></Link></dt>
<dd>
The context object provided to a [`@dagster_type_loader`](#dagster.dagster_type_loader)-decorated function during execution.
Users should not construct this object directly.
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeLoaderContext.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1374' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeLoaderContext.job_def" class="hash-link"></a></Link></dt>
<dd>
The underlying job definition being executed.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeLoaderContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1384' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeLoaderContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The op for which type loading is occurring.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.DagsterTypeLoaderContext.resources'>`property` resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/system.py#L1368' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.DagsterTypeLoaderContext.resources" class="hash-link"></a></Link></dt>
<dd>
The resources available to the type loader, specified by the <cite>required_resource_keys</cite> argument of the decorator.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.usable_as_dagster_type'>dagster.usable_as_dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/decorator.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.usable_as_dagster_type" class="hash-link"></a></Link></dt>
<dd>
Decorate a Python class to make it usable as a Dagster Type.
This is intended to make it straightforward to annotate existing business logic classes to
make them dagster types whose typecheck is an isinstance check against that python class.
Parameters:
- <strong>python_type</strong> (<em>cls</em>) – The python type to make usable as python type.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Name of the new Dagster type. If `None`, the name (`__name__`) of the `python_type` will be used.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A user-readable description of the type.
- <strong>loader</strong> (<em>Optional</em><em>[</em>[*DagsterTypeLoader*](#dagster.DagsterTypeLoader)<em>]</em>) – An instance of a class that inherits from [`DagsterTypeLoader`](#dagster.DagsterTypeLoader) and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the [`@dagster_type_loader`](#dagster.dagster_type_loader) decorator to construct these arguments.
Examples:
```python
# dagster_aws.s3.file_manager.S3FileHandle
@usable_as_dagster_type
class S3FileHandle(FileHandle):
def __init__(self, s3_bucket, s3_key):
self._s3_bucket = check.str_param(s3_bucket, 's3_bucket')
self._s3_key = check.str_param(s3_key, 's3_key')
@property
def s3_bucket(self):
return self._s3_bucket
@property
def s3_key(self):
return self._s3_key
@property
def path_desc(self):
return self.s3_path
@property
def s3_path(self):
return 's3://{bucket}/{key}'.format(bucket=self.s3_bucket, key=self.s3_key)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.make_python_type_usable_as_dagster_type'>dagster.make_python_type_usable_as_dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/types/dagster_type.py#L810' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.make_python_type_usable_as_dagster_type" class="hash-link"></a></Link></dt>
<dd>
Take any existing python type and map it to a dagster type (generally created with
[`DagsterType`](#dagster.DagsterType)) This can only be called once
on a given python type.
</dd>
</dl>
<div class="section" id="testing-types">
### Testing types
<dl>
<dt><Link class="anchor" id='dagster.check_dagster_type'>dagster.check_dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/dagster_type.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.check_dagster_type" class="hash-link"></a></Link></dt>
<dd>
Test a custom Dagster type.
Parameters:
- <strong>dagster_type</strong> (<em>Any</em>) – The Dagster type to test. Should be one of the [built-in types](#builtin)`built-in types`, a dagster type explicitly constructed with `as_dagster_type()`, `@usable_as_dagster_type`, or [`PythonObjectDagsterType()`](#dagster.PythonObjectDagsterType), or a Python type.
- <strong>value</strong> (<em>Any</em>) – The runtime value to test.
Returns: The result of the type check.Return type: [TypeCheck](ops.mdx#dagster.TypeCheck)
Examples:
```python
assert check_dagster_type(Dict[Any, Any], {'foo': 'bar'}).success
```
</dd>
</dl>
</div></div></div>
---
---
title: 'utilities'
sidebar_position: 1000
title_meta: 'utilities API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'utilities Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="utilities">
# Utilities
<dl>
<dt><Link class="anchor" id='dagster.file_relative_path'>dagster.file_relative_path <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/__init__.py#L119' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.file_relative_path" class="hash-link"></a></Link></dt>
<dd>
Get a path relative to the currently executing Python file.
This function is useful when one needs to load a file that is relative to the position of
the current file. (Such as when you encode a configuration file path in source file and want
in runnable in any current working directory)
Parameters:
- <strong>dunderfile</strong> (<em>str</em>) – Should always be `__file__`.
- <strong>relative_path</strong> (<em>str</em>) – Path to get relative to the currently executing file.
<strong>Examples</strong>:
```python
file_relative_path(__file__, 'path/relative/to/file')
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.config_from_files'>dagster.config_from_files <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/utils.py#L238' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.config_from_files" class="hash-link"></a></Link></dt>
<dd>
Constructs run config from YAML files.
Parameters: <strong>config_files</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – List of paths or glob patterns for yaml files
to load and parse as the run config.Returns: A run config dictionary constructed from provided YAML files.Return type: Dict[str, Any]Raises:
- <strong>FileNotFoundError</strong> – When a config file produces no results
- [DagsterInvariantViolationError](errors.mdx#dagster.DagsterInvariantViolationError)<strong>DagsterInvariantViolationError</strong> – When one of the YAML files is invalid and has a parse error.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.config_from_pkg_resources'>dagster.config_from_pkg_resources <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/utils.py#L301' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.config_from_pkg_resources" class="hash-link"></a></Link></dt>
<dd>
Load a run config from a package resource, using `pkg_resources.resource_string()`.
Example:
```python
config_from_pkg_resources(
pkg_resource_defs=[
('dagster_examples.airline_demo.environments', 'local_base.yaml'),
('dagster_examples.airline_demo.environments', 'local_warehouse.yaml'),
],
)
```
Parameters: <strong>pkg_resource_defs</strong> (<em>List</em><em>[</em><em>(</em><em>str</em><em>, </em><em>str</em><em>)</em><em>]</em>) – List of pkg_resource modules/files to
load as the run config.Returns: A run config dictionary constructed from the provided yaml stringsReturn type: Dict[Str, Any]Raises: [DagsterInvariantViolationError](errors.mdx#dagster.DagsterInvariantViolationError)<strong>DagsterInvariantViolationError</strong> – When one of the YAML documents is invalid and has a
parse error.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.config_from_yaml_strings'>dagster.config_from_yaml_strings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/utils.py#L276' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.config_from_yaml_strings" class="hash-link"></a></Link></dt>
<dd>
Static constructor for run configs from YAML strings.
Parameters: <strong>yaml_strings</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – List of yaml strings to parse as the run config.Returns: A run config dictionary constructed from the provided yaml stringsReturn type: Dict[Str, Any]Raises: [DagsterInvariantViolationError](errors.mdx#dagster.DagsterInvariantViolationError)<strong>DagsterInvariantViolationError</strong> – When one of the YAML documents is invalid and has a
parse error.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.get_dagster_logger'>dagster.get_dagster_logger <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/log.py#L137' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.get_dagster_logger" class="hash-link"></a></Link></dt>
<dd>
Creates a python logger whose output messages will be captured and converted into Dagster log
messages. This means they will have structured information such as the step_key, run_id, etc.
embedded into them, and will show up in the Dagster event log.
This can be used as a more convenient alternative to <cite>context.log</cite> in most cases. If log level
is not set explicitly, defaults to DEBUG.
Parameters: <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – If supplied, will create a logger with the name “dagster.builtin.\{name}”,
with properties inherited from the base Dagster logger. If omitted, the returned logger
will be named “dagster.builtin”.Returns: A logger whose output will be captured by Dagster.Return type: `logging.Logger`
Example:
```python
from dagster import get_dagster_logger, op
@op
def hello_op():
log = get_dagster_logger()
for i in range(5):
# do something
log.info(f"Did {i+1} things!")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.PreviewWarning'>`class` dagster.PreviewWarning <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-shared/dagster_shared/utils/warnings.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.PreviewWarning" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.BetaWarning'>`class` dagster.BetaWarning <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-shared/dagster_shared/utils/warnings.py#L44' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.BetaWarning" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.SupersessionWarning'>`class` dagster.SupersessionWarning <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-shared/dagster_shared/utils/warnings.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.SupersessionWarning" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster.make_email_on_run_failure_sensor'>dagster.make_email_on_run_failure_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/alert.py#L82' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster.make_email_on_run_failure_sensor" class="hash-link"></a></Link></dt>
<dd>
Create a job failure sensor that sends email via the SMTP protocol.
Parameters:
- <strong>email_from</strong> (<em>str</em>) – The sender email address to send the message from.
- <strong>email_password</strong> (<em>str</em>) – The password of the sender.
- <strong>email_to</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – The receipt email addresses to send the message to.
- <strong>email_body_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*RunFailureSensorContext*](schedules-sensors.mdx#dagster.RunFailureSensorContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the `RunFailureSensorContext` outputs the email body you want to send. Defaults to the plain text that contains error message, job name, and run ID.
- <strong>email_subject_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*RunFailureSensorContext*](schedules-sensors.mdx#dagster.RunFailureSensorContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the `RunFailureSensorContext` outputs the email subject you want to send. Defaults to “Dagster Run Failed: \<job_name>”.
- <strong>smtp_host</strong> (<em>str</em>) – The hostname of the SMTP server. Defaults to “smtp.gmail.com”.
- <strong>smtp_type</strong> (<em>str</em>) – The protocol; either “SSL” or “STARTTLS”. Defaults to SSL.
- <strong>smtp_port</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The SMTP port. Defaults to 465 for SSL, 587 for STARTTLS.
- <strong>smtp_user</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The SMTP user for authenticatication in the SMTP server. Defaults to the value of email_from.
- <strong>name</strong> – (Optional[str]): The name of the sensor. Defaults to “email_on_job_failure”.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your dagster-webserver instance. Specify this to allow messages to include deeplinks to the failed run.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*RepositorySelector*](schedules-sensors.mdx#dagster.RepositorySelector)<em>, </em>[*JobSelector*](schedules-sensors.mdx#dagster.JobSelector)<em>]</em><em>]</em><em>]</em>) – The jobs that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the repository fails. To monitor jobs in external repositories, use RepositorySelector and JobSelector.
- <strong>monitor_all_code_locations</strong> (<em>bool</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>job_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](graphs.mdx#dagster.GraphDefinition)<em>, </em>[*JobDefinition*](jobs.mdx#dagster.JobDefinition)<em>, </em>[*RepositorySelector*](schedules-sensors.mdx#dagster.RepositorySelector)<em>, </em>[*JobSelector*](schedules-sensors.mdx#dagster.JobSelector)<em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitored_jobs) The jobs that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the repository fails.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from the Dagster UI or via the GraphQL API.
- <strong>monitor_all_repositories</strong> (<em>bool</em>) – <span className="flag flag-warning">deprecated</span> If set to True, the sensor will monitor all runs in the Dagster instance. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
Examples:
```python
email_on_run_failure = make_email_on_run_failure_sensor(
email_from="no-reply@example.com",
email_password=os.getenv("ALERT_EMAIL_PASSWORD"),
email_to=["xxx@example.com"],
)
@repository
def my_repo():
return [my_job + email_on_run_failure]
```
```python
def my_message_fn(context: RunFailureSensorContext) -> str:
return (
f"Job {context.dagster_run.job_name} failed!"
f"Error: {context.failure_event.message}"
)
email_on_run_failure = make_email_on_run_failure_sensor(
email_from="no-reply@example.com",
email_password=os.getenv("ALERT_EMAIL_PASSWORD"),
email_to=["xxx@example.com"],
email_body_fn=my_message_fn,
email_subject_fn=lambda _: "Dagster Alert",
webserver_base_url="http://mycoolsite.com",
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster._utils.forked_pdb.ForkedPdb'>`class` dagster._utils.forked_pdb.ForkedPdb <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_utils/forked_pdb.py#L9' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster._utils.forked_pdb.ForkedPdb" class="hash-link"></a></Link></dt>
<dd>
A pdb subclass that may be used from a forked multiprocessing child.
<strong>Examples</strong>:
```python
from dagster._utils.forked_pdb import ForkedPdb
@solid
def complex_solid(_):
# some complicated stuff
ForkedPdb().set_trace()
# some other complicated stuff
```
You can initiate pipeline execution via the webserver and use the pdb debugger to examine/step through
execution at the breakpoint.
</dd>
</dl>
</div>
---
---
title: 'graphql (dagster-graphql)'
sidebar_position: 1000
title_meta: 'graphql (dagster-graphql) API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'graphql (dagster-graphql) Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="graphql-dagster-graphql">
# GraphQL (dagster-graphql)
<div class="section" id="python-client">
## Python client
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClient'>`class` dagster_graphql.DagsterGraphQLClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/client.py#L38' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClient" class="hash-link"></a></Link></dt>
<dd>
Official Dagster Python Client for GraphQL.
Utilizes the gql library to dispatch queries over HTTP to a remote Dagster GraphQL Server
As of now, all operations on this client are synchronous.
Intended usage:
```python
client = DagsterGraphQLClient("localhost", port_number=3000)
status = client.get_run_status(**SOME_RUN_ID**)
```
Parameters:
- <strong>hostname</strong> (<em>str</em>) – Hostname for the Dagster GraphQL API, like <cite>localhost</cite> or <cite>YOUR_ORG_HERE.dagster.cloud</cite>.
- <strong>port_number</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – Port number to connect to on the host. Defaults to None.
- <strong>transport</strong> (<em>Optional</em><em>[</em><em>Transport</em><em>]</em><em>, </em><em>optional</em>) – A custom transport to use to connect to the GraphQL API with (e.g. for custom auth). Defaults to None.
- <strong>use_https</strong> (<em>bool</em><em>, </em><em>optional</em>) – Whether to use https in the URL connection string for the GraphQL API. Defaults to False.
- <strong>timeout</strong> (<em>int</em>) – Number of seconds before requests should time out. Defaults to 60.
- <strong>headers</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Additional headers to include in the request. To use this client in Dagster Cloud, set the “Dagster-Cloud-Api-Token” header to a user token generated in the Dagster Cloud UI.
- <strong>path_prefix</strong> (<em>str</em>) – Optional path prefix for deployments behind a non-root path (e.g., `"/dagster"`). Must start with `"/"` and not end with `"/"` when non-empty. Defaults to `""`.
Raises: <strong>ConnectionError</strong> – if the client cannot connect to the host.
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClient.get_run_status'>get_run_status <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/client.py#L313' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClient.get_run_status" class="hash-link"></a></Link></dt>
<dd>
Get the status of a given Pipeline Run.
Parameters: <strong>run_id</strong> (<em>str</em>) – run id of the requested pipeline run.Raises:
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"PipelineNotFoundError"</strong><strong>, </strong><strong>message</strong><strong>)</strong> – if the requested run id is not found
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"PythonError"</strong><strong>, </strong><strong>message</strong><strong>)</strong> – on internal framework errors
Returns: returns a status Enum describing the state of the requested pipeline runReturn type: [DagsterRunStatus](../dagster/internals.mdx#dagster.DagsterRunStatus)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClient.reload_repository_location'>reload_repository_location <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/client.py#L339' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClient.reload_repository_location" class="hash-link"></a></Link></dt>
<dd>
Reloads a Dagster Repository Location, which reloads all repositories in that repository location.
This is useful in a variety of contexts, including refreshing the Dagster UI without restarting
the server.
Parameters: <strong>repository_location_name</strong> (<em>str</em>) – The name of the repository locationReturns: Object with information about the result of the reload requestReturn type: [ReloadRepositoryLocationInfo](#dagster_graphql.ReloadRepositoryLocationInfo)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClient.shutdown_repository_location'>shutdown_repository_location <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/client.py#L381' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClient.shutdown_repository_location" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 2.0.
:::
Shuts down the server that is serving metadata for the provided repository location.
This is primarily useful when you want the server to be restarted by the compute environment
in which it is running (for example, in Kubernetes, the pod in which the server is running
will automatically restart when the server is shut down, and the repository metadata will
be reloaded)
Parameters: <strong>repository_location_name</strong> (<em>str</em>) – The name of the repository locationReturns: Object with information about the result of the reload requestReturn type: ShutdownRepositoryLocationInfo
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClient.submit_job_execution'>submit_job_execution <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/client.py#L256' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClient.submit_job_execution" class="hash-link"></a></Link></dt>
<dd>
Submits a job with attached configuration for execution.
Parameters:
- <strong>job_name</strong> (<em>str</em>) – The job’s name
- <strong>repository_location_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the repository location where the job is located. If omitted, the client will try to infer the repository location from the available options on the Dagster deployment. Defaults to None.
- <strong>repository_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the repository where the job is located. If omitted, the client will try to infer the repository from the available options on the Dagster deployment. Defaults to None.
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*RunConfig*](../dagster/config.mdx#dagster.RunConfig)<em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em><em>]</em>) – This is the run config to execute the job with. Note that runConfigData is any-typed in the GraphQL type system. This type is used when passing in an arbitrary object for run config. However, it must conform to the constraints of the config schema for this job. If it does not, the client will throw a DagsterGraphQLClientError with a message of JobConfigValidationInvalid. Defaults to None.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A set of tags to add to the job execution.
- <strong>op_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of ops to execute.
- <strong>asset_selection</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>CoercibleToAssetKey</em><em>]</em><em>]</em>) – A list of asset keys to execute.
Raises:
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"InvalidStepError"</strong><strong>, </strong><strong>invalid_step_key</strong><strong>)</strong> – the job has an invalid step
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"InvalidOutputError"</strong><strong>, </strong><strong>body=error_object</strong><strong>)</strong> – some solid has an invalid output within the job. The error_object is of type dagster_graphql.InvalidOutputErrorInfo.
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"RunConflict"</strong><strong>, </strong><strong>message</strong><strong>)</strong> – a <cite>DagsterRunConflict</cite> occured during execution. This indicates that a conflicting job run already exists in run storage.
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"PipelineConfigurationInvalid"</strong><strong>, </strong><strong>invalid_step_key</strong><strong>)</strong> – the run_config is not in the expected format for the job
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"JobNotFoundError"</strong><strong>, </strong><strong>message</strong><strong>)</strong> – the requested job does not exist
- [DagsterGraphQLClientError](#dagster_graphql.DagsterGraphQLClientError)<strong>DagsterGraphQLClientError</strong><strong>(</strong><strong>"PythonError"</strong><strong>, </strong><strong>message</strong><strong>)</strong> – an internal framework error occurred
Returns: run id of the submitted pipeline runReturn type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.DagsterGraphQLClientError'>`exception` dagster_graphql.DagsterGraphQLClientError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/utils.py#L5' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.DagsterGraphQLClientError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.InvalidOutputErrorInfo'>`class` dagster_graphql.InvalidOutputErrorInfo <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/utils.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.InvalidOutputErrorInfo" class="hash-link"></a></Link></dt>
<dd>
This class gives information about an InvalidOutputError from submitting a pipeline for execution
from GraphQL.
Parameters:
- <strong>step_key</strong> (<em>str</em>) – key of the step that failed
- <strong>invalid_output_name</strong> (<em>str</em>) – the name of the invalid output from the given step
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.ReloadRepositoryLocationInfo'>`class` dagster_graphql.ReloadRepositoryLocationInfo <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/utils.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.ReloadRepositoryLocationInfo" class="hash-link"></a></Link></dt>
<dd>
This class gives information about the result of reloading
a Dagster repository location with a GraphQL mutation.
Parameters:
- <strong>status</strong> ([*ReloadRepositoryLocationStatus*](#dagster_graphql.ReloadRepositoryLocationStatus)) – The status of the reload repository location mutation
- <strong>failure_type</strong> – (Optional[str], optional): the failure type if <cite>status == ReloadRepositoryLocationStatus.FAILURE</cite>. Can be one of <cite>ReloadNotSupported</cite>, <cite>RepositoryLocationNotFound</cite>, or <cite>RepositoryLocationLoadFailure</cite>. Defaults to None.
- <strong>message</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – the failure message/reason if <cite>status == ReloadRepositoryLocationStatus.FAILURE</cite>. Defaults to None.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_graphql.ReloadRepositoryLocationStatus'>`class` dagster_graphql.ReloadRepositoryLocationStatus <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-graphql/dagster_graphql/client/utils.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_graphql.ReloadRepositoryLocationStatus" class="hash-link"></a></Link></dt>
<dd>
This enum describes the status of a GraphQL mutation to reload a Dagster repository location.
Parameters: <strong>Enum</strong> (<em>str</em>) – can be either <cite>ReloadRepositoryLocationStatus.SUCCESS</cite>
or <cite>ReloadRepositoryLocationStatus.FAILURE</cite>.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-airbyte library'
sidebar_position: 1000
title_meta: 'dagster-airbyte library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-airbyte library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-airbyte-library">
# dagster-airbyte library
This library provides a Dagster integration with [Airbyte](https://www.airbyte.com).
For more information on getting started, see the [Airbyte integration guide](https://docs.dagster.io/integrations/libraries/airbyte).
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteWorkspaceComponent'>`class` dagster_airbyte.AirbyteWorkspaceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/components/workspace_component/component.py#L230' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteWorkspaceComponent" class="hash-link"></a></Link></dt>
<dd>
Loads Airbyte connections from a given Airbyte workspace as Dagster assets.
Materializing these assets will trigger a sync of the Airbyte connection, enabling
you to schedule Airbyte syncs using Dagster.
Example:
```yaml
# defs.yaml
type: dagster_airbyte.AirbyteWorkspaceComponent
attributes:
workspace:
rest_api_base_url: http://localhost:8000/api/public/v1
configuration_api_base_url: http://localhost:8000/api/v1
workspace_id: your-workspace-id
client_id: "{{ env.AIRBYTE_CLIENT_ID }}"
client_secret: "{{ env.AIRBYTE_CLIENT_SECRET }}"
connection_selector:
by_name:
- my_postgres_to_snowflake_connection
- my_mysql_to_bigquery_connection
```
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteWorkspaceComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/components/workspace_component/component.py#L333' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteWorkspaceComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes an Airbyte sync for the selected connection.
This method can be overridden in a subclass to customize the sync execution behavior,
such as adding custom logging or handling sync results differently.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>airbyte</strong> – The BaseAirbyteWorkspace resource used to trigger and monitor syncs
Yields: AssetMaterialization or MaterializeResult events from the Airbyte sync
Example:
Override this method to add custom logging during sync execution:
```python
from dagster_airbyte import AirbyteWorkspaceComponent
import dagster as dg
class CustomAirbyteWorkspaceComponent(AirbyteWorkspaceComponent):
def execute(self, context, airbyte):
context.log.info(f"Starting Airbyte sync for connection")
yield from super().execute(context, airbyte)
context.log.info("Airbyte sync completed successfully")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteWorkspaceComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/components/workspace_component/component.py#L297' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteWorkspaceComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Airbyte connection table.
This method can be overridden in a subclass to customize how Airbyte connection tables
are converted to Dagster asset specs. By default, it delegates to the configured
DagsterAirbyteTranslator.
Parameters: <strong>props</strong> – The AirbyteConnectionTableProps containing information about the connection
and table/stream being syncedReturns: An AssetSpec that represents the Airbyte connection table as a Dagster asset
Example:
Override this method to add custom metadata to all Airbyte assets:
```python
from dagster_airbyte import AirbyteWorkspaceComponent
import dagster as dg
class CustomAirbyteWorkspaceComponent(AirbyteWorkspaceComponent):
def get_asset_spec(self, props):
base_spec = super().get_asset_spec(props)
return base_spec.replace_attributes(
metadata={
**base_spec.metadata,
"data_source": "airbyte",
"connection_id": props.connection_id
}
)
```
</dd>
</dl>
</dd>
</dl>
<div class="section" id="assets-airbyte-api">
### Assets (Airbyte API)
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteCloudWorkspace'>dagster_airbyte.AirbyteCloudWorkspace ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/resources.py#L832' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteCloudWorkspace" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource allows users to programatically interface with the Airbyte Cloud REST API to launch
syncs and monitor their progress for a given Airbyte Cloud workspace.
<strong>Examples:</strong>
```python
from dagster_airbyte import AirbyteCloudWorkspace, build_airbyte_assets_definitions
import dagster as dg
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
all_airbyte_assets = build_airbyte_assets_definitions(workspace=airbyte_workspace)
defs = dg.Definitions(
assets=all_airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteWorkspace'>dagster_airbyte.AirbyteWorkspace ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/resources.py#L712' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteWorkspace" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource allows users to programatically interface with the Airbyte REST API to launch
syncs and monitor their progress for a given Airbyte workspace.
<strong>Examples:</strong>
Using OAuth client credentials:
```python
import dagster as dg
from dagster_airbyte import AirbyteWorkspace, build_airbyte_assets_definitions
airbyte_workspace = AirbyteWorkspace(
rest_api_base_url=dg.EnvVar("AIRBYTE_REST_API_BASE_URL"),
configuration_api_base_url=dg.EnvVar("AIRBYTE_CONFIGURATION_API_BASE_URL"),
workspace_id=dg.EnvVar("AIRBYTE_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLIENT_SECRET"),
)
all_airbyte_assets = build_airbyte_assets_definitions(workspace=airbyte_workspace)
defs = dg.Definitions(
assets=all_airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
Using basic Authentication:
```python
import dagster as dg
from dagster_airbyte import AirbyteWorkspace, build_airbyte_assets_definitions
airbyte_workspace = AirbyteWorkspace(
rest_api_base_url=dg.EnvVar("AIRBYTE_REST_API_BASE_URL"),
configuration_api_base_url=dg.EnvVar("AIRBYTE_CONFIGURATION_API_BASE_URL"),
workspace_id=dg.EnvVar("AIRBYTE_WORKSPACE_ID"),
username=dg.EnvVar("AIRBYTE_USERNAME"),
password=dg.EnvVar("AIRBYTE_PASSWORD"),
)
all_airbyte_assets = build_airbyte_assets_definitions(workspace=airbyte_workspace)
defs = dg.Definitions(
assets=all_airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
Using no authentication:
```python
import dagster as dg
from dagster_airbyte import AirbyteWorkspace, build_airbyte_assets_definitions
airbyte_workspace = AirbyteWorkspace(
rest_api_base_url=dg.EnvVar("AIRBYTE_REST_API_BASE_URL"),
configuration_api_base_url=dg.EnvVar("AIRBYTE_CONFIGURATION_API_BASE_URL"),
workspace_id=dg.EnvVar("AIRBYTE_WORKSPACE_ID"),
)
all_airbyte_assets = build_airbyte_assets_definitions(workspace=airbyte_workspace)
defs = dg.Definitions(
assets=all_airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.DagsterAirbyteTranslator'>`class` dagster_airbyte.DagsterAirbyteTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/translator.py#L212' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.DagsterAirbyteTranslator" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Translator class which converts a <cite>AirbyteConnectionTableProps</cite> object into AssetSpecs.
Subclass this class to implement custom logic how to translate Airbyte content into asset spec.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.load_airbyte_asset_specs'>dagster_airbyte.load_airbyte_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/resources.py#L884' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.load_airbyte_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns a list of AssetSpecs representing the Airbyte content in the workspace.
Parameters:
- <strong>workspace</strong> (<em>BaseAirbyteWorkspace</em>) – The Airbyte workspace to fetch assets from.
- <strong>dagster_airbyte_translator</strong> (<em>Optional</em><em>[</em>[*DagsterAirbyteTranslator*](#dagster_airbyte.DagsterAirbyteTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Airbyte content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterAirbyteTranslator`](#dagster_airbyte.DagsterAirbyteTranslator).
- <strong>connection_selector_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnection</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that allows for filtering which Airbyte connection assets are created for.
Returns: The set of assets representing the Airbyte content in the workspace.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
Examples:
Loading the asset specs for a given Airbyte workspace:
```python
from dagster_airbyte import AirbyteWorkspace, load_airbyte_asset_specs
import dagster as dg
airbyte_workspace = AirbyteWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLIENT_SECRET"),
)
airbyte_specs = load_airbyte_asset_specs(airbyte_workspace)
dg.Definitions(assets=airbyte_specs)
```
Filter connections by name:
```python
from dagster_airbyte import AirbyteWorkspace, load_airbyte_asset_specs
import dagster as dg
airbyte_workspace = AirbyteWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLIENT_SECRET"),
)
airbyte_specs = load_airbyte_asset_specs(
workspace=airbyte_workspace,
connection_selector_fn=lambda connection: connection.name in ["connection1", "connection2"]
)
dg.Definitions(assets=airbyte_specs)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.load_airbyte_cloud_asset_specs'>dagster_airbyte.load_airbyte_cloud_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/resources.py#L962' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.load_airbyte_cloud_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use load_airbyte_asset_specs instead..
:::
Returns a list of AssetSpecs representing the Airbyte content in the workspace.
Parameters:
- <strong>workspace</strong> (<em>AirbyteCloudWorkspace</em>) – The Airbyte Cloud workspace to fetch assets from.
- <strong>dagster_airbyte_translator</strong> (<em>Optional</em><em>[</em>[*DagsterAirbyteTranslator*](#dagster_airbyte.DagsterAirbyteTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Airbyte content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterAirbyteTranslator`](#dagster_airbyte.DagsterAirbyteTranslator).
- <strong>connection_selector_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnection</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that allows for filtering which Airbyte connection assets are created for.
Returns: The set of assets representing the Airbyte content in the workspace.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
Examples:
Loading the asset specs for a given Airbyte Cloud workspace:
```python
from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specs
import dagster as dg
airbyte_cloud_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
airbyte_cloud_specs = load_airbyte_cloud_asset_specs(airbyte_cloud_workspace)
dg.Definitions(assets=airbyte_cloud_specs)
```
Filter connections by name:
```python
from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specs
import dagster as dg
airbyte_cloud_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
airbyte_cloud_specs = load_airbyte_cloud_asset_specs(
workspace=airbyte_cloud_workspace,
connection_selector_fn=lambda connection: connection.name in ["connection1", "connection2"]
)
dg.Definitions(assets=airbyte_cloud_specs)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.airbyte_assets'>@dagster_airbyte.airbyte_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/asset_decorator.py#L12' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.airbyte_assets" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Create a definition for how to sync the tables of a given Airbyte connection.
Parameters:
- <strong>connection_id</strong> (<em>str</em>) – The Airbyte Connection ID.
- <strong>workspace</strong> (<em>Union</em><em>[</em><em>AirbyteWorkspace</em><em>, </em><em>AirbyteCloudWorkspace</em><em>]</em>) – The Airbyte workspace to fetch assets from.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the op.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the asset group.
- <strong>dagster_airbyte_translator</strong> (<em>Optional</em><em>[</em>[*DagsterAirbyteTranslator*](#dagster_airbyte.DagsterAirbyteTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Airbyte content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterAirbyteTranslator`](#dagster_airbyte.DagsterAirbyteTranslator).
Examples:
Sync the tables of an Airbyte connection:
```python
from dagster_airbyte import AirbyteCloudWorkspace, airbyte_assets
import dagster as dg
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
@airbyte_assets(
connection_id="airbyte_connection_id",
workspace=airbyte_workspace,
)
def airbyte_connection_assets(context: dg.AssetExecutionContext, airbyte: AirbyteCloudWorkspace):
yield from airbyte.sync_and_poll(context=context)
defs = dg.Definitions(
assets=[airbyte_connection_assets],
resources={"airbyte": airbyte_workspace},
)
```
Sync the tables of an Airbyte connection with a custom translator:
```python
from dagster_airbyte import (
DagsterAirbyteTranslator,
AirbyteConnectionTableProps,
AirbyteCloudWorkspace,
airbyte_assets
)
import dagster as dg
class CustomDagsterAirbyteTranslator(DagsterAirbyteTranslator):
def get_asset_spec(self, props: AirbyteConnectionTableProps) -> dg.AssetSpec:
default_spec = super().get_asset_spec(props)
return default_spec.merge_attributes(
metadata={"custom": "metadata"},
)
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
@airbyte_assets(
connection_id="airbyte_connection_id",
workspace=airbyte_workspace,
dagster_airbyte_translator=CustomDagsterAirbyteTranslator()
)
def airbyte_connection_assets(context: dg.AssetExecutionContext, airbyte: AirbyteCloudWorkspace):
yield from airbyte.sync_and_poll(context=context)
defs = dg.Definitions(
assets=[airbyte_connection_assets],
resources={"airbyte": airbyte_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.build_airbyte_assets_definitions'>dagster_airbyte.build_airbyte_assets_definitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/asset_defs.py#L1047' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.build_airbyte_assets_definitions" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
The list of AssetsDefinition for all connections in the Airbyte workspace.
Parameters:
- <strong>workspace</strong> (<em>Union</em><em>[</em><em>AirbyteWorkspace</em><em>, </em><em>AirbyteCloudWorkspace</em><em>]</em>) – The Airbyte workspace to fetch assets from.
- <strong>dagster_airbyte_translator</strong> (<em>Optional</em><em>[</em>[*DagsterAirbyteTranslator*](#dagster_airbyte.DagsterAirbyteTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Airbyte content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterAirbyteTranslator`](#dagster_airbyte.DagsterAirbyteTranslator).
- <strong>connection_selector_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnection</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that allows for filtering which Airbyte connection assets are created for.
Returns: The list of AssetsDefinition for all connections in the Airbyte workspace.Return type: List[[AssetsDefinition](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)]
Examples:
Sync the tables of a Airbyte connection:
```python
from dagster_airbyte import AirbyteCloudWorkspace, build_airbyte_assets_definitions
import dagster as dg
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
airbyte_assets = build_airbyte_assets_definitions(workspace=workspace)
defs = dg.Definitions(
assets=airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
Sync the tables of a Airbyte connection with a custom translator:
```python
from dagster_airbyte import (
DagsterAirbyteTranslator,
AirbyteConnectionTableProps,
AirbyteCloudWorkspace,
build_airbyte_assets_definitions
)
import dagster as dg
class CustomDagsterAirbyteTranslator(DagsterAirbyteTranslator):
def get_asset_spec(self, props: AirbyteConnectionTableProps) -> dg.AssetSpec:
default_spec = super().get_asset_spec(props)
return default_spec.merge_attributes(
metadata={"custom": "metadata"},
)
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
airbyte_assets = build_airbyte_assets_definitions(
workspace=workspace,
dagster_airbyte_translator=CustomDagsterAirbyteTranslator()
)
defs = dg.Definitions(
assets=airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
Filter connections by name:
```python
from dagster_airbyte import AirbyteCloudWorkspace, build_airbyte_assets_definitions
import dagster as dg
airbyte_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
airbyte_assets = build_airbyte_assets_definitions(
workspace=workspace,
connection_selector_fn=lambda connection: connection.name in ["connection1", "connection2"]
)
defs = dg.Definitions(
assets=airbyte_assets,
resources={"airbyte": airbyte_workspace},
)
```
</dd>
</dl>
</div>
<div class="section" id="legacy">
### Legacy
<dl>
<dt><Link class="anchor" id='dagster_airbyte.AirbyteResource'>dagster_airbyte.AirbyteResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/legacy_resources.py#L262' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.AirbyteResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning superseded
This API has been superseded.
If you are using Airbyte 1.6.0 or higher, please see the migration guide: https://docs.dagster.io/integrations/libraries/airbyte/migration-guide.
:::
This resource allows users to programatically interface with the Airbyte REST API to launch
syncs and monitor their progress.
<strong>Examples:</strong>
```python
from dagster import job, EnvVar
from dagster_airbyte import AirbyteResource
my_airbyte_resource = AirbyteResource(
host=EnvVar("AIRBYTE_HOST"),
port=EnvVar("AIRBYTE_PORT"),
# If using basic auth
username=EnvVar("AIRBYTE_USERNAME"),
password=EnvVar("AIRBYTE_PASSWORD"),
)
airbyte_assets = build_airbyte_assets(
connection_id="87b7fe85-a22c-420e-8d74-b30e7ede77df",
destination_tables=["releases", "tags", "teams"],
)
Definitions(
assets=[airbyte_assets],
resources={"airbyte": my_airbyte_resource},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.load_assets_from_airbyte_instance'>dagster_airbyte.load_assets_from_airbyte_instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/asset_defs.py#L908' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.load_assets_from_airbyte_instance" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
If you are using Airbyte 1.6.0 or higher, please see the migration guide: https://docs.dagster.io/integrations/libraries/airbyte/migration-guide.
:::
Loads Airbyte connection assets from a configured AirbyteResource instance. This fetches information
about defined connections at initialization time, and will error on workspace load if the Airbyte
instance is not reachable.
Parameters:
- <strong>airbyte</strong> ([*ResourceDefinition*](../../../api/dagster/resources.mdx#dagster.ResourceDefinition)) – An AirbyteResource configured with the appropriate connection details.
- <strong>workspace_id</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The ID of the Airbyte workspace to load connections from. Only required if multiple workspaces exist in your instance.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>CoercibleToAssetKeyPrefix</em><em>]</em>) – A prefix for the asset keys created.
- <strong>create_assets_for_normalization_tables</strong> (<em>bool</em>) – If True, assets will be created for tables created by Airbyte’s normalization feature. If False, only the destination tables will be created. Defaults to True.
- <strong>connection_to_group_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Function which returns an asset group name for a given Airbyte connection name. If None, no groups will be created. Defaults to a basic sanitization function.
- <strong>connection_meta_to_group_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnectionMetadata</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Function which returns an asset group name for a given Airbyte connection metadata. If None and connection_to_group_fn is None, no groups will be created
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The I/O manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connection_to_io_manager_key_fn.
- <strong>connection_to_io_manager_key_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Function which returns an I/O manager key for a given Airbyte connection name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.
- <strong>connection_filter</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnectionMetadata</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – Optional function which takes in connection metadata and returns False if the connection should be excluded from the output assets.
- <strong>connection_to_asset_key_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnectionMetadata</em><em>, </em><em>str</em><em>]</em><em>, </em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – Optional function which takes in connection metadata and table name and returns an asset key for the table. If None, the default asset key is based on the table name. Any asset key prefix will be applied to the output of this function.
- <strong>connection_to_freshness_policy_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnectionMetadata</em><em>]</em><em>, </em><em>Optional</em><em>[</em>[*FreshnessPolicy*](../../../api/dagster/assets.mdx#dagster.FreshnessPolicy)<em>]</em><em>]</em><em>]</em>) – Optional function which takes in connection metadata and returns a freshness policy for the connection’s assets. If None, no freshness policies will be applied to the assets.
- <strong>connection_to_auto_materialize_policy_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>AirbyteConnectionMetadata</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>AutoMaterializePolicy</em><em>]</em><em>]</em><em>]</em>) – Optional function which takes in connection metadata and returns an auto materialization policy for the connection’s assets. If None, no auto materialization policies will be applied to the assets.
<strong>Examples:</strong>
Loading all Airbyte connections as assets:
```python
from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance
airbyte_instance = airbyte_resource.configured(
{
"host": "localhost",
"port": "8000",
}
)
airbyte_assets = load_assets_from_airbyte_instance(airbyte_instance)
```
Filtering the set of loaded connections:
```python
from dagster_airbyte import airbyte_resource, load_assets_from_airbyte_instance
airbyte_instance = airbyte_resource.configured(
{
"host": "localhost",
"port": "8000",
}
)
airbyte_assets = load_assets_from_airbyte_instance(
airbyte_instance,
connection_filter=lambda meta: "snowflake" in meta.name,
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.build_airbyte_assets'>dagster_airbyte.build_airbyte_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/asset_defs.py#L256' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.build_airbyte_assets" class="hash-link"></a></Link></dt>
<dd>
Builds a set of assets representing the tables created by an Airbyte sync operation.
Parameters:
- <strong>connection_id</strong> (<em>str</em>) – The Airbyte Connection ID that this op will sync. You can retrieve this value from the “Connections” tab of a given connector in the Airbyte UI.
- <strong>destination_tables</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – The names of the tables that you want to be represented in the Dagster asset graph for this sync. This will generally map to the name of the stream in Airbyte, unless a stream prefix has been specified in Airbyte.
- <strong>destination_database</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the destination database.
- <strong>destination_schema</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the destination schema.
- <strong>normalization_tables</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If you are using Airbyte’s normalization feature, you may specify a mapping of destination table to a list of derived tables that will be created by the normalization process.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A prefix for the asset keys inside this asset. If left blank, assets will have a key of <cite>AssetKey([table_name])</cite>.
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](../../../api/dagster/assets.mdx#dagster.SourceAsset)<em>, </em><em>str</em><em>, </em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>]</em><em>]</em><em>]</em>) – A list of assets to add as sources.
- <strong>upstream_assets</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – Deprecated, use deps instead. A list of assets to add as sources.
- <strong>stream_to_asset_map</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A mapping of an Airbyte stream name to a Dagster asset. This allows the use of the “prefix” setting in Airbyte with special characters that aren’t valid asset names.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airbyte.airbyte_sync_op'>dagster_airbyte.airbyte_sync_op `=` \<dagster._core.definitions.op_definition.OpDefinition object> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airbyte/dagster_airbyte/ops.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airbyte.airbyte_sync_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Executes a Airbyte job sync for a given `connection_id`, and polls until that sync
completes, raising an error if it is unsuccessful. It outputs a AirbyteOutput which contains
the job details for a given `connection_id`.
It requires the use of the `airbyte_resource`, which allows it to
communicate with the Airbyte API.
Examples:
```python
from dagster import job
from dagster_airbyte import airbyte_resource, airbyte_sync_op
my_airbyte_resource = airbyte_resource.configured(
{
"host": {"env": "AIRBYTE_HOST"},
"port": {"env": "AIRBYTE_PORT"},
}
)
sync_foobar = airbyte_sync_op.configured({"connection_id": "foobar"}, name="sync_foobar")
@job(resource_defs={"airbyte": my_airbyte_resource})
def my_simple_airbyte_job():
sync_foobar()
@job(resource_defs={"airbyte": my_airbyte_resource})
def my_composed_airbyte_job():
final_foobar_state = sync_foobar(start_after=some_op())
other_op(final_foobar_state)
```
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-airlift library'
sidebar_position: 1000
title_meta: 'dagster-airlift library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-airlift library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-airlift-library">
# dagster-airlift library
<div class="section" id="core-dagster-airlift-core">
## Core (dagster_airlift.core)
<div class="section" id="airflowinstance">
### AirflowInstance
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowInstance'>`class` dagster_airlift.core.AirflowInstance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_instance.py#L61' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowInstance" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A class that represents a running Airflow Instance and provides methods for interacting with its REST API.
Parameters:
- <strong>auth_backend</strong> ([*AirflowAuthBackend*](#dagster_airlift.core.AirflowAuthBackend)) – The authentication backend to use when making requests to the Airflow instance.
- <strong>name</strong> (<em>str</em>) – The name of the Airflow instance. This will be prefixed to any assets automatically created using this instance.
- <strong>batch_task_instance_limit</strong> (<em>int</em>) – The number of task instances to query at a time when fetching task instances. Defaults to 100.
- <strong>batch_dag_runs_limit</strong> (<em>int</em>) – The number of dag runs to query at a time when fetching dag runs. Defaults to 100.
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowInstance.get_run_state'>get_run_state <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_instance.py#L440' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowInstance.get_run_state" class="hash-link"></a></Link></dt>
<dd>
Given a run ID of an airflow dag, return the state of that run.
Parameters:
- <strong>dag_id</strong> (<em>str</em>) – The dag id.
- <strong>run_id</strong> (<em>str</em>) – The run id.
Returns: The state of the run. Will be one of the states defined by Airflow.Return type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowInstance.trigger_dag'>trigger_dag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_instance.py#L354' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowInstance.trigger_dag" class="hash-link"></a></Link></dt>
<dd>
Trigger a dag run for the given dag_id.
Does not wait for the run to finish. To wait for the completed run to finish, use [`wait_for_run_completion()`](#dagster_airlift.core.AirflowInstance.wait_for_run_completion).
Parameters:
- <strong>dag_id</strong> (<em>str</em>) – The dag id to trigger.
- <strong>logical_date</strong> (<em>Optional</em><em>[</em><em>datetime.datetime</em><em>]</em>) – The Airflow logical_date to use for the dag run. If not provided, the current time will be used. Previously known as execution_date in Airflow; find more information in the Airflow docs: [https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-does-execution-date-mean](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-does-execution-date-mean)
Returns: The dag run id.Return type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowInstance.wait_for_run_completion'>wait_for_run_completion <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_instance.py#L418' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowInstance.wait_for_run_completion" class="hash-link"></a></Link></dt>
<dd>
Given a run ID of an airflow dag, wait for that run to reach a completed state.
Parameters:
- <strong>dag_id</strong> (<em>str</em>) – The dag id.
- <strong>run_id</strong> (<em>str</em>) – The run id.
- <strong>timeout</strong> (<em>int</em>) – The number of seconds to wait before timing out.
Returns: None
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowAuthBackend'>`class` dagster_airlift.core.AirflowAuthBackend <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_instance.py#L41' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowAuthBackend" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
An abstract class that represents an authentication backend for an Airflow instance.
Requires two methods to be implemented by subclasses:
- get_session: Returns a requests.Session object that can be used to make requests to the Airflow instance, and handles authentication.
- get_webserver_url: Returns the base URL of the Airflow webserver.
The <cite>dagster-airlift</cite> package provides the following default implementations:
- `dagster-airlift.core.AirflowBasicAuthBackend`: An authentication backend that uses Airflow’s basic auth to authenticate with the Airflow instance.
- `dagster-airlift.mwaa.MwaaSessionAuthBackend`: An authentication backend that uses AWS MWAA’s web login token to authenticate with the Airflow instance (requires <cite>dagster-airlift[mwaa]</cite>).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowBasicAuthBackend'>`class` dagster_airlift.core.AirflowBasicAuthBackend <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/basic_auth.py#L7' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowBasicAuthBackend" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A [`dagster_airlift.core.AirflowAuthBackend`](#dagster_airlift.core.AirflowAuthBackend) that authenticates using basic auth.
Parameters:
- <strong>webserver_url</strong> (<em>str</em>) – The URL of the webserver.
- <strong>username</strong> (<em>str</em>) – The username to authenticate with.
- <strong>password</strong> (<em>str</em>) – The password to authenticate with.
Examples:
Creating a [`AirflowInstance`](#dagster_airlift.core.AirflowInstance) using this backend.
```python
from dagster_airlift.core import AirflowInstance, AirflowBasicAuthBackend
af_instance = AirflowInstance(
name="my-instance",
auth_backend=AirflowBasicAuthBackend(
webserver_url="https://my-webserver-hostname",
username="my-username",
password="my-password"
)
)
```
</dd>
</dl>
</div>
<div class="section" id="assets-definitions">
### Assets & Definitions
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.build_defs_from_airflow_instance'>dagster_airlift.core.build_defs_from_airflow_instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/load_defs.py#L77' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.build_defs_from_airflow_instance" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Builds a [`dagster.Definitions`](../../../api/dagster/definitions.mdx#dagster.Definitions) object from an Airflow instance.
For every DAG in the Airflow instance, this function will create a Dagster asset for the DAG
with an asset key instance_name/dag/dag_id. It will also create a sensor that polls the Airflow
instance for DAG runs and emits Dagster events for each successful run.
An optional <cite>defs</cite> argument can be provided, where the user can pass in a [`dagster.Definitions`](../../../api/dagster/definitions.mdx#dagster.Definitions)
object containing assets which are mapped to Airflow DAGs and tasks. These assets will be enriched with
metadata from the Airflow instance, and placed upstream of the automatically generated DAG assets.
An optional <cite>event_transformer_fn</cite> can be provided, which allows the user to modify the Dagster events
produced by the sensor. The function takes the Dagster events produced by the sensor and returns a sequence
of Dagster events.
An optional <cite>dag_selector_fn</cite> can be provided, which allows the user to filter which DAGs assets are created for.
The function takes a [`dagster_airlift.core.serialization.serialized_data.DagInfo`](#dagster_airlift.core.DagInfo) object and returns a
boolean indicating whether the DAG should be included.
Parameters:
- <strong>airflow_instance</strong> ([*AirflowInstance*](#dagster_airlift.core.AirflowInstance)) – The Airflow instance to build assets and the sensor from.
- <strong>defs</strong> – Optional[Definitions]: A [`dagster.Definitions`](../../../api/dagster/definitions.mdx#dagster.Definitions) object containing assets that are mapped to Airflow DAGs and tasks.
- <strong>sensor_minimum_interval_seconds</strong> (<em>int</em>) – The minimum interval in seconds between sensor runs.
- <strong>event_transformer_fn</strong> (<em>DagsterEventTransformerFn</em>) – A function that allows for modifying the Dagster events produced by the sensor.
- <strong>dag_selector_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em>[*DagInfo*](#dagster_airlift.core.DagInfo)<em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – A function that allows for filtering which DAGs assets are created for.
- <strong>source_code_retrieval_enabled</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether to retrieve source code for the Airflow DAGs. By default, source code is retrieved when the number of DAGs is under 50 for performance reasons. This setting overrides the default behavior.
- <strong>default_sensor_status</strong> (<em>Optional</em><em>[</em><em>DefaultSensorStatus</em><em>]</em>) – The default status for the sensor. By default, the sensor will be enabled.
Returns: A [`dagster.Definitions`](../../../api/dagster/definitions.mdx#dagster.Definitions) object containing the assets and sensor.Return type: [Definitions](../../../api/dagster/definitions.mdx#dagster.Definitions)
Examples:
Building a [`dagster.Definitions`](../../../api/dagster/definitions.mdx#dagster.Definitions) object from an Airflow instance.
```python
from dagster_airlift.core import (
AirflowInstance,
AirflowBasicAuthBackend,
build_defs_from_airflow_instance,
)
from .constants import AIRFLOW_BASE_URL, AIRFLOW_INSTANCE_NAME, PASSWORD, USERNAME
airflow_instance = AirflowInstance(
auth_backend=AirflowBasicAuthBackend(
webserver_url=AIRFLOW_BASE_URL, username=USERNAME, password=PASSWORD
),
name=AIRFLOW_INSTANCE_NAME,
)
defs = build_defs_from_airflow_instance(airflow_instance=airflow_instance)
```
Providing task-mapped assets to the function.
```python
from dagster import Definitions
from dagster_airlift.core import (
AirflowInstance,
AirflowBasicAuthBackend,
assets_with_task_mappings,
build_defs_from_airflow_instance,
)
...
defs = build_defs_from_airflow_instance(
airflow_instance=airflow_instance, # same as above
defs=Definitions(
assets=assets_with_task_mappings(
dag_id="rebuild_iris_models",
task_mappings={
"my_task": [AssetSpec("my_first_asset"), AssetSpec("my_second_asset")],
},
),
),
)
```
Providing a custom event transformer function.
```python
from typing import Sequence
from dagster import Definitions, SensorEvaluationContext
from dagster_airlift.core import (
AirflowInstance,
AirflowBasicAuthBackend,
AssetEvent,
assets_with_task_mappings,
build_defs_from_airflow_instance,
AirflowDefinitionsData,
)
...
def add_tags_to_events(
context: SensorEvaluationContext,
defs_data: AirflowDefinitionsData,
events: Sequence[AssetEvent]
) -> Sequence[AssetEvent]:
altered_events = []
for event in events:
altered_events.append(event._replace(tags={"my_tag": "my_value"}))
return altered_events
defs = build_defs_from_airflow_instance(
airflow_instance=airflow_instance, # same as above
event_transformer_fn=add_tags_to_events,
)
```
Filtering which DAGs assets are created for.
```python
from dagster import Definitions
from dagster_airlift.core import (
AirflowInstance,
AirflowBasicAuthBackend,
AssetEvent,
assets_with_task_mappings,
build_defs_from_airflow_instance,
DagInfo,
)
...
def only_include_dag(dag_info: DagInfo) -> bool:
return dag_info.dag_id == "my_dag_id"
defs = build_defs_from_airflow_instance(
airflow_instance=airflow_instance, # same as above
dag_selector_fn=only_include_dag,
)
```
</dd>
</dl>
<div class="section" id="mapping-dagster-assets-to-airflow-tasks-dags">
#### Mapping Dagster assets to Airflow tasks/DAGs
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.assets_with_task_mappings'>dagster_airlift.core.assets_with_task_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/top_level_dag_def_api.py#L61' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.assets_with_task_mappings" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Modify assets to be associated with a particular task in Airlift tooling.
Used in concert with <cite>build_defs_from_airflow_instance</cite> to observe an airflow
instance to monitor the tasks that are associated with the assets and
keep their materialization histories up to date.
Concretely this adds metadata to all asset specs in the provided definitions
with the provided dag_id and task_id. The dag_id comes from the dag_id argument;
the task_id comes from the key of the provided task_mappings dictionary.
There is a single metadata key “airlift/task-mapping” that is used to store
this information. It is a list of dictionaries with keys “dag_id” and “task_id”.
Example:
```python
from dagster import AssetSpec, Definitions, asset
from dagster_airlift.core import assets_with_task_mappings
@asset
def asset_one() -> None: ...
Definitions(
assets=assets_with_task_mappings(
dag_id="dag_one",
task_mappings={
"task_one": [asset_one],
"task_two": [AssetSpec(key="asset_two"), AssetSpec(key="asset_three")],
},
)
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.assets_with_dag_mappings'>dagster_airlift.core.assets_with_dag_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/top_level_dag_def_api.py#L108' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.assets_with_dag_mappings" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Modify assets to be associated with a particular dag in Airlift tooling.
Used in concert with <cite>build_defs_from_airflow_instance</cite> to observe an airflow
instance to monitor the dags that are associated with the assets and
keep their materialization histories up to date.
In contrast with <cite>assets_with_task_mappings</cite>, which maps assets on a per-task basis, this is used in concert with
<cite>proxying_to_dagster</cite> dag-level mappings where an entire dag is migrated at once.
Concretely this adds metadata to all asset specs in the provided definitions
with the provided dag_id. The dag_id comes from the key of the provided dag_mappings dictionary.
There is a single metadata key “airlift/dag-mapping” that is used to store
this information. It is a list of strings, where each string is a dag_id which the asset is associated with.
Example:
```python
from dagster import AssetSpec, Definitions, asset
from dagster_airlift.core import assets_with_dag_mappings
@asset
def asset_one() -> None: ...
Definitions(
assets=assets_with_dag_mappings(
dag_mappings={
"dag_one": [asset_one],
"dag_two": [AssetSpec(key="asset_two"), AssetSpec(key="asset_three")],
},
)
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.assets_with_multiple_task_mappings'>dagster_airlift.core.assets_with_multiple_task_mappings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/multiple_tasks.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.assets_with_multiple_task_mappings" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Given an asset or assets definition, return a new asset or assets definition with metadata
that indicates that it is targeted by multiple airflow tasks. An example of this would
be a separate weekly and daily dag that contains a task that targets a single asset.
```python
from dagster import Definitions, AssetSpec, asset
from dagster_airlift import (
build_defs_from_airflow_instance,
targeted_by_multiple_tasks,
assets_with_task_mappings,
)
# Asset maps to a single task.
@asset
def other_asset(): ...
# Asset maps to a physical entity which is produced by two different airflow tasks.
@asset
def scheduled_twice(): ...
defs = build_defs_from_airflow_instance(
airflow_instance=airflow_instance,
defs=Definitions(
assets=[
*assets_with_task_mappings(
dag_id="other_dag",
task_mappings={
"task1": [other_asset]
},
),
*assets_with_multiple_task_mappings(
assets=[scheduled_twice],
task_handles=[
{"dag_id": "weekly_dag", "task_id": "task1"},
{"dag_id": "daily_dag", "task_id": "task1"},
],
),
]
),
)
```
</dd>
</dl>
</div>
<div class="section" id="annotations-for-customizable-components">
#### Annotations for customizable components
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.DagSelectorFn'>dagster_airlift.core.DagSelectorFn <a href='https://github.com/dagster-io/dagster/blob/master/uv/python/versions/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/typing.py#L1285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.DagSelectorFn" class="hash-link"></a></Link></dt>
<dd>
alias of `Callable`[[[`DagInfo`](#dagster_airlift.core.DagInfo)], `bool`]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.DagsterEventTransformerFn'>dagster_airlift.core.DagsterEventTransformerFn <a href='https://github.com/dagster-io/dagster/blob/master/uv/python/versions/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/typing.py#L1285' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.DagsterEventTransformerFn" class="hash-link"></a></Link></dt>
<dd>
alias of `Callable`[[`SensorEvaluationContext`, [`AirflowDefinitionsData`](#dagster_airlift.core.AirflowDefinitionsData), `Sequence`[[`AssetMaterialization`](../../../api/dagster/ops.mdx#dagster.AssetMaterialization)]], `Iterable`[[`AssetMaterialization`](../../../api/dagster/ops.mdx#dagster.AssetMaterialization) | [`AssetObservation`](../../../api/dagster/assets.mdx#dagster.AssetObservation) | `AssetCheckEvaluation`]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.TaskHandleDict'>`class` dagster_airlift.core.TaskHandleDict <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/multiple_tasks.py#L17' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.TaskHandleDict" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
</dd>
</dl>
</div>
<div class="section" id="objects-for-retrieving-information-about-the-airflow-dagster-mapping">
#### Objects for retrieving information about the Airflow/Dagster mapping
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.DagInfo'>`class` dagster_airlift.core.DagInfo <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/serialization/serialized_data.py#L42' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.DagInfo" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A record containing information about a given airflow dag.
Users should not instantiate this class directly. It is provided when customizing which DAGs are included
in the generated definitions using the <cite>dag_selector_fn</cite> argument of [`build_defs_from_airflow_instance()`](#dagster_airlift.core.build_defs_from_airflow_instance).
Parameters: <strong>metadata</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The metadata associated with the dag, retrieved by the Airflow REST API:
[https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dags](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dags)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowDefinitionsData'>`class` dagster_airlift.core.AirflowDefinitionsData <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_defs_data.py#L44' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowDefinitionsData" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A class that holds data about the assets that are mapped to Airflow dags and tasks, and
provides methods for retrieving information about the mappings.
The user should not instantiate this class directly. It is provided when customizing the events
that are generated by the Airflow sensor using the <cite>event_transformer_fn</cite> argument of
[`build_defs_from_airflow_instance()`](#dagster_airlift.core.build_defs_from_airflow_instance).
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowDefinitionsData.asset_keys_in_task'>asset_keys_in_task <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_defs_data.py#L170' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowDefinitionsData.asset_keys_in_task" class="hash-link"></a></Link></dt>
<dd>
Returns the asset keys that are mapped to the given task.
Parameters:
- <strong>dag_id</strong> (<em>str</em>) – The dag id.
- <strong>task_id</strong> (<em>str</em>) – The task id.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowDefinitionsData.task_ids_in_dag'>task_ids_in_dag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_defs_data.py#L107' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowDefinitionsData.task_ids_in_dag" class="hash-link"></a></Link></dt>
<dd>
Returns the task ids within the given dag_id.
Parameters: <strong>dag_id</strong> (<em>str</em>) – The dag id.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.AirflowDefinitionsData.instance_name'>`property` instance_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/airflow_defs_data.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.AirflowDefinitionsData.instance_name" class="hash-link"></a></Link></dt>
<dd>
The name of the Airflow instance.
</dd>
</dl>
</dd>
</dl>
</div></div>
<div class="section" id="airflowinstancecomponent">
### AirflowInstanceComponent
<dl>
<dt><Link class="anchor" id='dagster_airlift.core.components.AirflowInstanceComponent'>`class` dagster_airlift.core.components.AirflowInstanceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/core/components/airflow_instance/component.py#L206' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.core.components.AirflowInstanceComponent" class="hash-link"></a></Link></dt>
<dd>
Loads Airflow DAGs and tasks from an Airflow instance as Dagster assets.
This component connects to an Airflow instance, retrieves metadata about DAGs and tasks,
and creates corresponding Dagster assets. It supports mapping Airflow tasks to existing
Dagster assets or creating new assets to represent Airflow workflows.
Example:
```yaml
# defs.yaml
type: dagster_airlift.core.AirflowInstanceComponent
attributes:
name: my_airflow_instance
auth:
type: basic_auth
webserver_url: "{{ env.AIRFLOW_WEBSERVER_URL }}"
username: "{{ env.AIRFLOW_USERNAME }}"
password: "{{ env.AIRFLOW_PASSWORD }}"
filter:
dag_id_ilike: "analytics_%"
retrieve_datasets: true
```
</dd>
</dl>
</div>
<div class="section" id="mwaa-dagster-airlift-mwaa">
### MWAA (dagster_airlift.mwaa)
<dl>
<dt><Link class="anchor" id='dagster_airlift.mwaa.MwaaSessionAuthBackend'>`class` dagster_airlift.mwaa.MwaaSessionAuthBackend <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/mwaa/auth.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.mwaa.MwaaSessionAuthBackend" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A [`dagster_airlift.core.AirflowAuthBackend`](#dagster_airlift.core.AirflowAuthBackend) that authenticates to AWS MWAA.
Under the hood, this class uses the MWAA boto3 session to request a web login token and then
uses the token to authenticate to the MWAA web server.
Parameters:
- <strong>mwaa_session</strong> (<em>boto3.Session</em>) – The boto3 MWAA session
- <strong>env_name</strong> (<em>str</em>) – The name of the MWAA environment
Examples:
Creating an AirflowInstance pointed at a MWAA environment.
```python
import boto3
from dagster_airlift.mwaa import MwaaSessionAuthBackend
from dagster_airlift.core import AirflowInstance
boto_client = boto3.client("mwaa")
af_instance = AirflowInstance(
name="my-mwaa-instance",
auth_backend=MwaaSessionAuthBackend(
mwaa_client=boto_client,
env_name="my-mwaa-env"
)
)
```
</dd>
</dl>
</div>
<div class="section" id="in-airflow-dagster-airlift-in-airflow">
### In Airflow (dagster_airlift.in_airflow)
<div class="section" id="proxying">
#### Proxying
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.proxying_to_dagster'>dagster_airlift.in_airflow.proxying_to_dagster <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/proxying_fn.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.proxying_to_dagster" class="hash-link"></a></Link></dt>
<dd>
Proxies tasks and dags to Dagster based on provided proxied state.
Expects a dictionary of in-scope global variables to be provided (typically retrieved with <cite>globals()</cite>), and a proxied state dictionary
(typically retrieved with [`load_proxied_state_from_yaml()`](#dagster_airlift.in_airflow.load_proxied_state_from_yaml)) for dags in that global state. This function will modify in-place the
dictionary of global variables to replace proxied tasks with appropriate Dagster operators.
In the case of task-level proxying, the proxied tasks will be replaced with new operators that are constructed by the provided <cite>build_from_task_fn</cite>.
A default implementation of this function is provided in <cite>DefaultProxyTaskToDagsterOperator</cite>.
In the case of dag-level proxying, the entire dag structure will be replaced with a single task that is constructed by the provided <cite>build_from_dag_fn</cite>.
A default implementation of this function is provided in <cite>DefaultProxyDAGToDagsterOperator</cite>.
Parameters:
- <strong>global_vars</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The global variables in the current context. In most cases, retrieved with <cite>globals()</cite> (no import required). This is equivalent to what airflow already does to introspect the dags which exist in a given module context: [https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#loading-dags](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#loading-dags)
- <strong>proxied_state</strong> (<em>AirflowMigrationState</em>) – The proxied state for the dags.
- <strong>logger</strong> (<em>Optional</em><em>[</em><em>logging.Logger</em><em>]</em>) – The logger to use. Defaults to logging.getLogger(“dagster_airlift”).
Examples:
Typical usage of this function is to be called at the end of a dag file, retrieving proxied_state from an accompanying <cite>proxied_state</cite> path.
```python
from pathlib import Path
from airflow import DAG
from airflow.operators.python import PythonOperator
from dagster._time import get_current_datetime_midnight
from dagster_airlift.in_airflow import proxying_to_dagster
from dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yaml
with DAG(
dag_id="daily_interval_dag",
...,
) as minute_dag:
PythonOperator(task_id="my_task", python_callable=...)
# At the end of the dag file, so we can ensure dags are loaded into globals.
proxying_to_dagster(
proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"),
global_vars=globals(),
)
```
You can also provide custom implementations of the <cite>build_from_task_fn</cite> function to customize the behavior of task-level proxying.
```python
from dagster_airlift.in_airflow import proxying_to_dagster, BaseProxyTaskToDagsterOperator
from airflow.models.operator import BaseOperator
... # Dag code here
class CustomAuthTaskProxyOperator(BaseProxyTaskToDagsterOperator):
def get_dagster_session(self, context: Context) -> requests.Session:
# Add custom headers to the session
return requests.Session(headers={"Authorization": "Bearer my_token"})
def get_dagster_url(self, context: Context) -> str:
# Use a custom environment variable for the dagster url
return os.environ["CUSTOM_DAGSTER_URL"]
@classmethod
def build_from_task(cls, task: BaseOperator) -> "CustomAuthTaskProxyOperator":
# Custom logic to build the operator from the task (task_id should remain the same)
if task.task_id == "my_task_needs_more_retries":
return CustomAuthTaskProxyOperator(task_id=task_id, retries=3)
else:
return CustomAuthTaskProxyOperator(task_id=task_id)
proxying_to_dagster(
proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"),
global_vars=globals(),
build_from_task_fn=CustomAuthTaskProxyOperator.build_from_task,
)
```
You can do the same for dag-level proxying by providing a custom implementation of the <cite>build_from_dag_fn</cite> function.
```python
from dagster_airlift.in_airflow import proxying_to_dagster, BaseProxyDAGToDagsterOperator
from airflow.models.dag import DAG
... # Dag code here
class CustomAuthDAGProxyOperator(BaseProxyDAGToDagsterOperator):
def get_dagster_session(self, context: Context) -> requests.Session:
# Add custom headers to the session
return requests.Session(headers={"Authorization": "Bearer my_token"})
def get_dagster_url(self, context: Context) -> str:
# Use a custom environment variable for the dagster url
return os.environ["CUSTOM_DAGSTER_URL"]
@classmethod
def build_from_dag(cls, dag: DAG) -> "CustomAuthDAGProxyOperator":
# Custom logic to build the operator from the dag (DAG id should remain the same)
if dag.dag_id == "my_dag_needs_more_retries":
return CustomAuthDAGProxyOperator(task_id="custom override", retries=3, dag=dag)
else:
return CustomAuthDAGProxyOperator(task_id="basic_override", dag=dag)
proxying_to_dagster(
proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"),
global_vars=globals(),
build_from_dag_fn=CustomAuthDAGProxyOperator.build_from_dag,
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.BaseDagsterAssetsOperator'>`class` dagster_airlift.in_airflow.BaseDagsterAssetsOperator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/base_asset_operator.py#L41' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.BaseDagsterAssetsOperator" class="hash-link"></a></Link></dt>
<dd>
Interface for an operator which materializes dagster assets.
This operator needs to implement the following methods:
>
- get_dagster_session: Returns a requests session that can be used to make requests to the Dagster API.
- get_dagster_url: Returns the URL for the Dagster instance.
- filter_asset_nodes: Filters asset nodes (which are returned from Dagster’s graphql API) to only include those
Optionally, these methods can be overridden as well:
>
- get_partition_key: Determines the partition key to use to trigger the dagster run. This method will only be
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.load_proxied_state_from_yaml'>dagster_airlift.in_airflow.load_proxied_state_from_yaml <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/proxied_state.py#L152' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.load_proxied_state_from_yaml" class="hash-link"></a></Link></dt>
<dd>
Loads the proxied state from a directory of yaml files.
Expects the directory to contain yaml files, where each file corresponds to the id of a dag (ie: <cite>dag_id.yaml</cite>).
This directory is typically constructed using the <cite>dagster-airlift</cite> CLI:
>
```bash
AIRFLOW_HOME=... dagster-airlift proxy scaffold
```
The file should have either of the following structure.
In the case of task-level proxying:
>
```yaml
tasks:
- id: task_id
proxied: true
- id: task_id
proxied: false
```
In the case of dag-level proxying:
>
```yaml
proxied: true
```
Parameters: <strong>proxied_yaml_path</strong> (<em>Path</em>) – The path to the directory containing the yaml files.Returns: The proxied state of the dags and tasks in Airflow.Return type: [AirflowProxiedState](#dagster_airlift.in_airflow.AirflowProxiedState)
</dd>
</dl>
<div class="section" id="proxying-state">
##### Proxying state
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.AirflowProxiedState'>`class` dagster_airlift.in_airflow.AirflowProxiedState <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/proxied_state.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.AirflowProxiedState" class="hash-link"></a></Link></dt>
<dd>
A class to store the proxied state of dags and tasks in Airflow.
Typically, this is constructed by [`load_proxied_state_from_yaml()`](#dagster_airlift.in_airflow.load_proxied_state_from_yaml).
Parameters: <strong>dags</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em>[*DagProxiedState*](#dagster_airlift.in_airflow.DagProxiedState)<em>]</em>) – A dictionary of dag_id to DagProxiedState.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.DagProxiedState'>`class` dagster_airlift.in_airflow.DagProxiedState <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/proxied_state.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.DagProxiedState" class="hash-link"></a></Link></dt>
<dd>
A class to store the proxied state of tasks in a dag.
Parameters:
- <strong>tasks</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em>[*TaskProxiedState*](#dagster_airlift.in_airflow.TaskProxiedState)<em>]</em>) – A dictionary of task_id to TaskProxiedState. If the entire dag is proxied, or proxied state is not set for a task, the task_id will not be present in this dictionary.
- <strong>proxied</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – A boolean indicating whether the entire dag is proxied. If this is None, then the dag proxies at the task level (or
- <strong>all</strong><strong>)</strong><strong>.</strong> (<em>proxying state has not been set at</em>)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.TaskProxiedState'>`class` dagster_airlift.in_airflow.TaskProxiedState <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/proxied_state.py#L8' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.TaskProxiedState" class="hash-link"></a></Link></dt>
<dd>
A class to store the proxied state of a task.
Parameters:
- <strong>task_id</strong> (<em>str</em>) – The id of the task.
- <strong>proxied</strong> (<em>bool</em>) – A boolean indicating whether the task is proxied.
</dd>
</dl>
</div>
<div class="section" id="task-level-proxying">
##### Task-level proxying
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.BaseProxyTaskToDagsterOperator'>`class` dagster_airlift.in_airflow.BaseProxyTaskToDagsterOperator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/task_proxy_operator.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.BaseProxyTaskToDagsterOperator" class="hash-link"></a></Link></dt>
<dd>
An operator that proxies task execution to Dagster assets with metadata that map to this task’s dag ID and task ID.
For the DAG ID and task ID that this operator proxies, it expects there to be corresponding assets
in the linked Dagster deployment that have metadata entries with the key <cite>dagster-airlift/task-mapping</cite> that
map to this DAG ID and task ID. This metadata is typically set using the
[`dagster_airlift.core.assets_with_task_mappings()`](#dagster_airlift.core.assets_with_task_mappings) function.
The following methods must be implemented by subclasses:
>
- `get_dagster_session()` (inherited from [`BaseDagsterAssetsOperator`](#dagster_airlift.in_airflow.BaseDagsterAssetsOperator))
- `get_dagster_url()` (inherited from [`BaseDagsterAssetsOperator`](#dagster_airlift.in_airflow.BaseDagsterAssetsOperator))
- `build_from_task()` A class method which takes the task to be proxied, and constructs
There is a default implementation of this operator, [`DefaultProxyTaskToDagsterOperator`](#dagster_airlift.in_airflow.DefaultProxyTaskToDagsterOperator),
which is used by [`proxying_to_dagster()`](#dagster_airlift.in_airflow.proxying_to_dagster) if no override operator is provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.DefaultProxyTaskToDagsterOperator'>`class` dagster_airlift.in_airflow.DefaultProxyTaskToDagsterOperator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/task_proxy_operator.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.DefaultProxyTaskToDagsterOperator" class="hash-link"></a></Link></dt>
<dd>
The default task proxying operator - which opens a blank session and expects the dagster URL to be set in the environment.
The dagster url is expected to be set in the environment as DAGSTER_URL.
This operator should not be instantiated directly - it is instantiated by [`proxying_to_dagster()`](#dagster_airlift.in_airflow.proxying_to_dagster) if no
override operator is provided.
</dd>
</dl>
</div>
<div class="section" id="dag-level-proxying">
##### DAG-level proxying
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.BaseProxyDAGToDagsterOperator'>`class` dagster_airlift.in_airflow.BaseProxyDAGToDagsterOperator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/dag_proxy_operator.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.BaseProxyDAGToDagsterOperator" class="hash-link"></a></Link></dt>
<dd>
An operator base class that proxies the entire DAG’s execution to Dagster assets with
metadata that map to the DAG id used by this task.
For the Dag ID that this operator proxies, it expects there to be corresponding assets
in the linked Dagster deployment that have metadata entries with the key <cite>dagster-airlift/dag-mapping</cite> that
map to this Dag ID. This metadata is typically set using the
[`dagster_airlift.core.assets_with_dag_mappings()`](#dagster_airlift.core.assets_with_dag_mappings) function.
The following methods must be implemented by subclasses:
>
- `get_dagster_session()` (inherited from [`BaseDagsterAssetsOperator`](#dagster_airlift.in_airflow.BaseDagsterAssetsOperator))
- `get_dagster_url()` (inherited from [`BaseDagsterAssetsOperator`](#dagster_airlift.in_airflow.BaseDagsterAssetsOperator))
- `build_from_dag()` A class method which takes the DAG to be proxied, and constructs
There is a default implementation of this operator, [`DefaultProxyDAGToDagsterOperator`](#dagster_airlift.in_airflow.DefaultProxyDAGToDagsterOperator),
which is used by [`proxying_to_dagster()`](#dagster_airlift.in_airflow.proxying_to_dagster) if no override operator is provided.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_airlift.in_airflow.DefaultProxyDAGToDagsterOperator'>`class` dagster_airlift.in_airflow.DefaultProxyDAGToDagsterOperator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-airlift/dagster_airlift/in_airflow/dag_proxy_operator.py#L62' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_airlift.in_airflow.DefaultProxyDAGToDagsterOperator" class="hash-link"></a></Link></dt>
<dd>
The default task proxying operator - which opens a blank session and expects the dagster URL to be set in the environment.
The dagster url is expected to be set in the environment as DAGSTER_URL.
This operator should not be instantiated directly - it is instantiated by [`proxying_to_dagster()`](#dagster_airlift.in_airflow.proxying_to_dagster) if no
override operator is provided.
</dd>
</dl>
</div></div></div></div></div>
---
---
title: 'dagster-aws library'
sidebar_position: 1000
title_meta: 'dagster-aws library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-aws library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-aws-library">
# dagster-aws library
Utilities for interfacing with AWS with Dagster.
<div class="section" id="s3">
## S3
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3Resource'>dagster_aws.s3.S3Resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/resources.py#L53' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.S3Resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource that gives access to S3.
The underlying S3 session is created by calling
`boto3.session.Session(profile_name)`.
The returned resource object is an S3 client, an instance of <cite>botocore.client.S3</cite>.
Example:
```python
from dagster import job, op, Definitions
from dagster_aws.s3 import S3Resource
@op
def example_s3_op(s3: S3Resource):
return s3.get_client().list_objects_v2(
Bucket='my-bucket',
Prefix='some-key'
)
@job
def example_job():
example_s3_op()
Definitions(
jobs=[example_job],
resources={'s3': S3Resource(region_name='us-west-1')}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3PickleIOManager'>dagster_aws.s3.S3PickleIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py#L88' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.S3PickleIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using S3 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for S3 and the backing bucket.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
```python
from dagster import asset, Definitions
from dagster_aws.s3 import S3PickleIOManager, S3Resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": S3PickleIOManager(
s3_resource=S3Resource(),
s3_bucket="my-cool-bucket",
s3_prefix="my-cool-prefix",
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3ComputeLogManager'>`class` dagster_aws.s3.S3ComputeLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/compute_log_manager.py#L32' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.S3ComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
Logs compute function stdout and stderr to S3.
Users should not instantiate this class directly. Instead, use a YAML block in `dagster.yaml`
such as the following:
```YAML
compute_logs:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
bucket: "mycorp-dagster-compute-logs"
local_dir: "/tmp/cool"
prefix: "dagster-test-"
use_ssl: true
verify: true
verify_cert_path: "/path/to/cert/bundle.pem"
endpoint_url: "http://alternate-s3-host.io"
skip_empty_files: true
upload_interval: 30
upload_extra_args:
ServerSideEncryption: "AES256"
show_url_only: false
region: "us-west-1"
```
Parameters:
- <strong>bucket</strong> (<em>str</em>) – The name of the s3 bucket to which to log.
- <strong>local_dir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Path to the local directory in which to stage logs. Default: `dagster_shared.seven.get_system_temp_directory()`.
- <strong>prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Prefix for the log file keys.
- <strong>use_ssl</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to use SSL. Default True.
- <strong>verify</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether or not to verify SSL certificates. Default True.
- <strong>verify_cert_path</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A filename of the CA cert bundle to use. Only used if <cite>verify</cite> set to False.
- <strong>endpoint_url</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Override for the S3 endpoint url.
- <strong>skip_empty_files</strong> – (Optional[bool]): Skip upload of empty log files.
- <strong>upload_interval</strong> – (Optional[int]): Interval in seconds to upload partial log files to S3. By default, will only upload when the capture is complete.
- <strong>upload_extra_args</strong> – (Optional[dict]): Extra args for S3 file upload
- <strong>show_url_only</strong> – (Optional[bool]): Only show the URL of the log file in the UI, instead of fetching and displaying the full content. Default False.
- <strong>region</strong> – (Optional[str]): The region of the S3 bucket. If not specified, will use the default region of the AWS session.
- <strong>inst_data</strong> (<em>Optional</em><em>[</em>[*ConfigurableClassData*](../../../api/dagster/internals.mdx#dagster._serdes.ConfigurableClassData)<em>]</em>) – Serializable representation of the compute log manager when newed up from config.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3Coordinate'>dagster_aws.s3.S3Coordinate DagsterType<a href="#dagster_aws.s3.S3Coordinate" class="hash-link"></a></Link></dt>
<dd>
A [`dagster.DagsterType`](../../../api/dagster/types.mdx#dagster.DagsterType) intended to make it easier to pass information about files on S3
from op to op. Objects of this type should be dicts with `'bucket'` and `'key'` keys,
and may be hydrated from config in the intuitive way, e.g., for an input with the name
`s3_file`:
```YAML
inputs:
s3_file:
value:
bucket: my-bucket
key: my-key
```
</dd>
</dl>
<div class="section" id="file-manager">
### File manager
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3FileHandle'>`class` dagster_aws.s3.S3FileHandle <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/file_manager.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.S3FileHandle" class="hash-link"></a></Link></dt>
<dd>
A reference to a file on S3.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.S3FileManagerResource'>dagster_aws.s3.S3FileManagerResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/resources.py#L180' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.S3FileManagerResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for Dagster resources that utilize structured config.
This class is a subclass of both `ResourceDefinition` and `Config`.
Example definition:
```python
class WriterResource(ConfigurableResource):
prefix: str
def output(self, text: str) -> None:
print(f"{self.prefix}{text}")
```
Example usage:
```python
@asset
def asset_that_uses_writer(writer: WriterResource):
writer.output("text")
defs = Definitions(
assets=[asset_that_uses_writer],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
You can optionally use this class to model configuration only and vend an object
of a different type for use at runtime. This is useful for those who wish to
have a separate object that manages configuration and a separate object at runtime. Or
where you want to directly use a third-party class that you do not control.
To do this you override the <cite>create_resource</cite> methods to return a different object.
```python
class WriterResource(ConfigurableResource):
prefix: str
def create_resource(self, context: InitResourceContext) -> Writer:
# Writer is pre-existing class defined else
return Writer(self.prefix)
```
Example usage:
```python
@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
writer.output("text")
defs = Definitions(
assets=[use_preexisting_writer_as_resource],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
</dd>
</dl>
</div></div>
<div class="section" id="ecs">
## ECS
<dl>
<dt><Link class="anchor" id='dagster_aws.ecs.EcsRunLauncher'>dagster_aws.ecs.EcsRunLauncher RunLauncher <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ecs.EcsRunLauncher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
RunLauncher that starts a task in ECS for each Dagster job run.
Parameters:
- <strong>inst_data</strong> (<em>Optional</em><em>[</em>[*ConfigurableClassData*](../../../api/dagster/internals.mdx#dagster._serdes.ConfigurableClassData)<em>]</em>) – If not provided, defaults to None.
- <strong>task_definition</strong> – If not provided, defaults to None.
- <strong>container_name</strong> (<em>str</em>) – If not provided, defaults to “run”.
- <strong>secrets</strong> (<em>Optional</em><em>[</em><em>list</em><em>[</em><em>str</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>secrets_tag</strong> (<em>str</em>) – If not provided, defaults to “dagster”.
- <strong>env_vars</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>include_sidecars</strong> (<em>bool</em>) – If not provided, defaults to False.
- <strong>use_current_ecs_task_config</strong> (<em>bool</em>) – If not provided, defaults to True.
- <strong>run_task_kwargs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>run_resources</strong> (<em>Optional</em><em>[</em><em>dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>run_ecs_tags</strong> (<em>Optional</em><em>[</em><em>list</em><em>[</em><em>dict</em><em>[</em><em>str</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>propagate_tags</strong> (<em>Optional</em><em>[</em><em>dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – If not provided, defaults to None.
- <strong>task_definition_prefix</strong> (<em>str</em>) – If not provided, defaults to “run”.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.ecs.ecs_executor'>dagster_aws.ecs.ecs_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ecs/executor.py#L79' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ecs.ecs_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Executor which launches steps as ECS tasks.
To use the <cite>ecs_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster_aws.ecs import ecs_executor
from dagster import job, op
@op(
tags={"ecs/cpu": "256", "ecs/memory": "512"},
)
def ecs_op():
pass
@job(executor_def=ecs_executor)
def ecs_job():
ecs_op()
```
Then you can configure the executor with run config as follows:
```YAML
execution:
config:
cpu: 1024
memory: 2048
ephemeral_storage: 10
task_overrides:
containerOverrides:
- name: run
environment:
- name: MY_ENV_VAR
value: "my_value"
```
<cite>max_concurrent</cite> limits the number of ECS tasks that will execute concurrently for one run. By default
there is no limit- it will maximally parallel as allowed by the DAG. Note that this is not a
global limit.
Configuration set on the ECS tasks created by the <cite>ECSRunLauncher</cite> will also be
set on the tasks created by the <cite>ecs_executor</cite>.
Configuration set using <cite>tags</cite> on a <cite>@job</cite> will only apply to the <cite>run</cite> level. For configuration
to apply at each <cite>step</cite> it must be set using <cite>tags</cite> for each <cite>@op</cite>.
</dd>
</dl>
</div>
<div class="section" id="rds">
## RDS
<dl>
<dt><Link class="anchor" id='dagster_aws.rds.RDSResource'>dagster_aws.rds.RDSResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/rds/resources.py#L13' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.rds.RDSResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
A resource for interacting with the AWS RDS service.
It wraps both the AWS RDS client ([https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds.html)),
and the AWS RDS Data client ([https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds-data.html](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rds-data.html)).
The AWS-RDS client (`RDSResource.get_rds_client()`) allows access to the management layer of RDS (creating, starting, configuring databases).
The AWS RDS Data (`RDSResource.get_data_client`) allows executing queries on the SQL databases themselves.
Note that AWS RDS Data service is only available for Aurora database. For accessing data from other types of RDS databases,
you should directly use the corresponding SQL client instead (e.g. Postgres/MySQL).
Example:
```python
from dagster import Definitions, asset
from dagster_aws.rds import RDSResource
@asset
def my_table(rds_resource: RDSResource):
with rds_resource.get_rds_client() as rds_client:
rds_client.describe_db_instances()['DBInstances']
with rds_resource.get_data_client() as data_client:
data_client.execute_statement(
resourceArn="RESOURCE_ARN",
secretArn="SECRET_ARN",
sql="SELECT * from mytable",
)
Definitions(
assets=[my_table],
resources={
"rds_resource": RDSResource(
region_name="us-west-1"
)
}
)
```
</dd>
</dl>
</div>
<div class="section" id="redshift">
## Redshift
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.RedshiftClientResource'>dagster_aws.redshift.RedshiftClientResource ResourceDefinition<a href="#dagster_aws.redshift.RedshiftClientResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource enables connecting to a Redshift cluster and issuing queries against that
cluster.
Example:
```python
from dagster import Definitions, asset, EnvVar
from dagster_aws.redshift import RedshiftClientResource
@asset
def example_redshift_asset(context, redshift: RedshiftClientResource):
redshift.get_client().execute_query('SELECT 1', fetch_results=True)
redshift_configured = RedshiftClientResource(
host='my-redshift-cluster.us-east-1.redshift.amazonaws.com',
port=5439,
user='dagster',
password=EnvVar("DAGSTER_REDSHIFT_PASSWORD"),
database='dev',
)
Definitions(
assets=[example_redshift_asset],
resources={'redshift': redshift_configured},
)
```
</dd>
</dl>
<div class="section" id="testing">
### Testing
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.FakeRedshiftClientResource'>dagster_aws.redshift.FakeRedshiftClientResource ResourceDefinition<a href="#dagster_aws.redshift.FakeRedshiftClientResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource enables connecting to a Redshift cluster and issuing queries against that
cluster.
Example:
```python
from dagster import Definitions, asset, EnvVar
from dagster_aws.redshift import RedshiftClientResource
@asset
def example_redshift_asset(context, redshift: RedshiftClientResource):
redshift.get_client().execute_query('SELECT 1', fetch_results=True)
redshift_configured = RedshiftClientResource(
host='my-redshift-cluster.us-east-1.redshift.amazonaws.com',
port=5439,
user='dagster',
password=EnvVar("DAGSTER_REDSHIFT_PASSWORD"),
database='dev',
)
Definitions(
assets=[example_redshift_asset],
resources={'redshift': redshift_configured},
)
```
</dd>
</dl>
</div></div>
<div class="section" id="emr">
## EMR
<dl>
<dt><Link class="anchor" id='dagster_aws.emr.emr_pyspark_step_launcher'>dagster_aws.emr.emr_pyspark_step_launcher ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark_step_launcher.py#L35' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.emr.emr_pyspark_step_launcher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning superseded
This API has been superseded.
While there is no plan to remove this functionality, for new projects, we recommend using Dagster Pipes. For more information, see https://docs.dagster.io/integrations/external-pipelines.
:::
- <strong>spark_config</strong>:
- <strong>cluster_id</strong>: Name of the job flow (cluster) on which to execute.
- <strong>region_name</strong>: The AWS region that the cluster is in.
- <strong>action_on_failure</strong>: The EMR action to take when the cluster step fails: [https://docs.aws.amazon.com/emr/latest/APIReference/API_StepConfig.html](https://docs.aws.amazon.com/emr/latest/APIReference/API_StepConfig.html)
- <strong>staging_bucket</strong>: S3 bucket to use for passing files between the plan process and EMR process.
- <strong>staging_prefix</strong>: S3 key prefix inside the staging_bucket to use for files passed the plan process and EMR process
- <strong>wait_for_logs</strong>: If set, the system will wait for EMR logs to appear on S3. Note that logs are copied every 5 minutes, so enabling this will add several minutes to the job runtime.
- <strong>local_job_package_path</strong>: Absolute path to the package that contains the job definition(s) whose steps will execute remotely on EMR. This is a path on the local fileystem of the process executing the job. The expectation is that this package will also be available on the python path of the launched process running the Spark step on EMR, either deployed on step launch via the deploy_local_job_package option, referenced on s3 via the s3_job_package_path option, or installed on the cluster via bootstrap actions.
- <strong>local_pipeline_package_path</strong>: (legacy) Absolute path to the package that contains the pipeline definition(s) whose steps will execute remotely on EMR. This is a path on the local fileystem of the process executing the pipeline. The expectation is that this package will also be available on the python path of the launched process running the Spark step on EMR, either deployed on step launch via the deploy_local_pipeline_package option, referenced on s3 via the s3_pipeline_package_path option, or installed on the cluster via bootstrap actions.
- <strong>deploy_local_job_package</strong>: If set, before every step run, the launcher will zip up all the code in local_job_package_path, upload it to s3, and pass it to spark-submit’s –py-files option. This gives the remote process access to up-to-date user code. If not set, the assumption is that some other mechanism is used for distributing code to the EMR cluster. If this option is set to True, s3_job_package_path should not also be set.
- <strong>deploy_local_pipeline_package</strong>: (legacy) If set, before every step run, the launcher will zip up all the code in local_job_package_path, upload it to s3, and pass it to spark-submit’s –py-files option. This gives the remote process access to up-to-date user code. If not set, the assumption is that some other mechanism is used for distributing code to the EMR cluster. If this option is set to True, s3_job_package_path should not also be set.
- <strong>s3_job_package_path</strong>: If set, this path will be passed to the –py-files option of spark-submit. This should usually be a path to a zip file. If this option is set, deploy_local_job_package should not be set to True.
- <strong>s3_pipeline_package_path</strong>: If set, this path will be passed to the –py-files option of spark-submit. This should usually be a path to a zip file. If this option is set, deploy_local_pipeline_package should not be set to True.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.emr.EmrJobRunner'>`class` dagster_aws.emr.EmrJobRunner <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/emr.py#L49' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.emr.EmrJobRunner" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.emr.EmrError'>`class` dagster_aws.emr.EmrError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/emr.py#L45' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.emr.EmrError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.emr.EmrClusterState'>dagster_aws.emr.EmrClusterState `=` \<enum 'EmrClusterState'> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/types.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.emr.EmrClusterState" class="hash-link"></a></Link></dt>
<dd>
Cluster state for EMR.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.emr.EmrStepState'>dagster_aws.emr.EmrStepState `=` \<enum 'EmrStepState'> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/types.py#L31' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.emr.EmrStepState" class="hash-link"></a></Link></dt>
<dd>
Step state for EMR.
</dd>
</dl>
</div>
<div class="section" id="cloudwatch">
## CloudWatch
<dl>
<dt><Link class="anchor" id='dagster_aws.cloudwatch.cloudwatch_logger'>dagster_aws.cloudwatch.cloudwatch_logger LoggerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.cloudwatch.cloudwatch_logger" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Core class for defining loggers.
Loggers are job-scoped logging handlers, which will be automatically invoked whenever
dagster messages are logged from within a job.
Parameters:
- <strong>logger_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitLoggerContext*](../../../api/dagster/loggers.mdx#dagster.InitLoggerContext)<em>]</em><em>, </em><em>logging.Logger</em><em>]</em>) – User-provided function to instantiate the logger. This logger will be automatically invoked whenever the methods on `context.log` are called from within job compute logic.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](../../../api/dagster/config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this logger.
</dd>
</dl>
</div>
<div class="section" id="secretsmanager">
## SecretsManager
Resources which surface SecretsManager secrets for use in Dagster resources and jobs.
<dl>
<dt><Link class="anchor" id='dagster_aws.secretsmanager.SecretsManagerResource'>dagster_aws.secretsmanager.SecretsManagerResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/secretsmanager/resources.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.secretsmanager.SecretsManagerResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that gives access to AWS SecretsManager.
The underlying SecretsManager session is created by calling
`boto3.session.Session(profile_name)`.
The returned resource object is a SecretsManager client, an instance of <cite>botocore.client.SecretsManager</cite>.
Example:
```python
from dagster import build_op_context, job, op
from dagster_aws.secretsmanager import SecretsManagerResource
@op
def example_secretsmanager_op(secretsmanager: SecretsManagerResource):
return secretsmanager.get_client().get_secret_value(
SecretId='arn:aws:secretsmanager:region:aws_account_id:secret:appauthexample-AbCdEf'
)
@job
def example_job():
example_secretsmanager_op()
Definitions(
jobs=[example_job],
resources={
'secretsmanager': SecretsManagerResource(
region_name='us-west-1'
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.secretsmanager.SecretsManagerSecretsResource'>dagster_aws.secretsmanager.SecretsManagerSecretsResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/secretsmanager/resources.py#L135' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.secretsmanager.SecretsManagerSecretsResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that provides a dict which maps selected SecretsManager secrets to
their string values. Also optionally sets chosen secrets as environment variables.
Example:
```python
import os
from dagster import build_op_context, job, op, ResourceParam
from dagster_aws.secretsmanager import SecretsManagerSecretsResource
@op
def example_secretsmanager_secrets_op(secrets: SecretsManagerSecretsResource):
return secrets.fetch_secrets().get("my-secret-name")
@op
def example_secretsmanager_secrets_op_2(secrets: SecretsManagerSecretsResource):
with secrets.secrets_in_environment():
return os.getenv("my-other-secret-name")
@job
def example_job():
example_secretsmanager_secrets_op()
example_secretsmanager_secrets_op_2()
Definitions(
jobs=[example_job],
resources={
'secrets': SecretsManagerSecretsResource(
region_name='us-west-1',
secrets_tag="dagster",
add_to_environment=True,
)
}
)
```
Note that your ops must also declare that they require this resource with or it will not be initialized
for the execution of their compute functions.
</dd>
</dl>
</div>
<div class="section" id="ssm">
## SSM
<dl>
<dt><Link class="anchor" id='dagster_aws.ssm.SSMResource'>dagster_aws.ssm.SSMResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ssm/resources.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ssm.SSMResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that gives access to AWS Systems Manager Parameter Store.
The underlying Parameter Store session is created by calling
`boto3.session.Session(profile_name)`.
The returned resource object is a Systems Manager client, an instance of <cite>botocore.client.ssm</cite>.
Example:
```python
from typing import Any
from dagster import build_op_context, job, op
from dagster_aws.ssm import SSMResource
@op
def example_ssm_op(ssm: SSMResource):
return ssm.get_client().get_parameter(
Name="a_parameter"
)
@job
def example_job():
example_ssm_op()
Definitions(
jobs=[example_job],
resources={
'ssm': SSMResource(
region_name='us-west-1'
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.ssm.ParameterStoreResource'>dagster_aws.ssm.ParameterStoreResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ssm/resources.py#L145' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ssm.ParameterStoreResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that provides a dict which maps selected SSM Parameter Store parameters to
their string values. Optionally sets selected parameters as environment variables.
Example:
```python
import os
from typing import Dict
from dagster import build_op_context, job, op
from dagster_aws.ssm import ParameterStoreResource, ParameterStoreTag
@op
def example_parameter_store_op(parameter_store: ParameterStoreResource):
return parameter_store.fetch_parameters().get("my-parameter-name")
@op
def example_parameter_store_op_2(parameter_store: ParameterStoreResource):
with parameter_store.parameters_in_environment():
return os.getenv("my-other-parameter-name")
@job
def example_job():
example_parameter_store_op()
example_parameter_store_op_2()
defs = Definitions(
jobs=[example_job],
resource_defs={
'parameter_store': ParameterStoreResource(
region_name='us-west-1',
parameter_tags=[ParameterStoreTag(key='my-tag-key', values=['my-tag-value'])],
add_to_environment=True,
with_decryption=True,
)
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.ssm.ParameterStoreTag'>`class` dagster_aws.ssm.ParameterStoreTag <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ssm/resources.py#L139' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ssm.ParameterStoreTag" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
</dd>
</dl>
</div>
<div class="section" id="pipes">
## Pipes
<div class="section" id="context-injectors">
### Context injectors
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesS3ContextInjector'>`class` dagster_aws.pipes.PipesS3ContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/context_injectors.py#L20' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesS3ContextInjector" class="hash-link"></a></Link></dt>
<dd>
A context injector that injects context by writing to a temporary S3 location.
Parameters:
- <strong>bucket</strong> (<em>str</em>) – The S3 bucket to write to.
- <strong>client</strong> (<em>S3Client</em>) – A boto3 client to use to write to S3.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the S3 key. Defaults to a random string.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesLambdaEventContextInjector'>`class` dagster_aws.pipes.PipesLambdaEventContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/context_injectors.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesLambdaEventContextInjector" class="hash-link"></a></Link></dt>
<dd>
Injects context via AWS Lambda event input.
Should be paired with :py:class`~dagster_pipes.PipesMappingParamsLoader` on the Lambda side.
</dd>
</dl>
</div>
<div class="section" id="message-readers">
### Message readers
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesS3MessageReader'>`class` dagster_aws.pipes.PipesS3MessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/message_readers.py#L100' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesS3MessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by periodically reading message chunks from a specified S3
bucket.
If <cite>log_readers</cite> is passed, this reader will also start the passed readers
when the first message is received from the external process.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a chunk
- <strong>bucket</strong> (<em>str</em>) – The S3 bucket to read from.
- <strong>client</strong> (<em>boto3.client</em>) – A boto3 S3 client.
- <strong>log_readers</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>PipesLogReader</em><em>]</em><em>]</em>) – A set of log readers for logs on S3.
- <strong>include_stdio_in_messages</strong> (<em>bool</em>) – Whether to send stdout/stderr to Dagster via Pipes messages. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesCloudWatchMessageReader'>`class` dagster_aws.pipes.PipesCloudWatchMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/message_readers.py#L344' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesCloudWatchMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that consumes AWS CloudWatch logs to read pipes messages.
</dd>
</dl>
</div>
<div class="section" id="clients">
### Clients
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesLambdaClient'>`class` dagster_aws.pipes.PipesLambdaClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/lambda_.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesLambdaClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for invoking AWS lambda.
By default context is injected via the lambda input event and messages are parsed out of the
4k tail of logs.
Parameters:
- <strong>client</strong> (<em>boto3.client</em>) – The boto lambda client used to call invoke.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the lambda function. Defaults to [`PipesLambdaEventContextInjector`](#dagster_aws.pipes.PipesLambdaEventContextInjector).
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the lambda function. Defaults to `PipesLambdaLogsMessageReader`.
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesLambdaClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/lambda_.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesLambdaClient.run" class="hash-link"></a></Link></dt>
<dd>
Synchronously invoke a lambda function, enriched with the pipes protocol.
Parameters:
- <strong>function_name</strong> (<em>str</em>) – The name of the function to use.
- <strong>event</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A JSON serializable object to pass as input to the lambda.
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesGlueClient'>`class` dagster_aws.pipes.PipesGlueClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/glue.py#L28' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesGlueClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for invoking AWS Glue jobs.
Parameters:
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the Glue job, for example, [`PipesS3ContextInjector`](#dagster_aws.pipes.PipesS3ContextInjector).
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the glue job run. Defaults to `PipesCloudWatchsMessageReader`. When provided with [`PipesCloudWatchMessageReader`](#dagster_aws.pipes.PipesCloudWatchMessageReader), it will be used to recieve logs and events from the `.../output/\<job-run-id>` CloudWatch log stream created by AWS Glue. Note that AWS Glue routes both `stderr` and `stdout` from the main job process into this LogStream.
- <strong>client</strong> (<em>Optional</em><em>[</em><em>boto3.client</em><em>]</em>) – The boto Glue client used to launch the Glue job
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the Glue job run when the Dagster process receives a termination signal.
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesGlueClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/glue.py#L60' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesGlueClient.run" class="hash-link"></a></Link></dt>
<dd>
Start a Glue job, enriched with the pipes protocol.
See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/StartJobRun)
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>start_job_run_params</strong> (<em>Dict</em>) – Parameters for the `start_job_run` boto3 Glue client call.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional Dagster metadata to pass to the Glue job.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesECSClient'>`class` dagster_aws.pipes.PipesECSClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/ecs.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesECSClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for running AWS ECS tasks.
Parameters:
- <strong>client</strong> (<em>Any</em>) – The boto ECS client used to launch the ECS task
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the ECS task. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the ECS task. Defaults to [`PipesCloudWatchMessageReader`](#dagster_aws.pipes.PipesCloudWatchMessageReader).
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the ECS task when the Dagster process receives a termination signal.
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesECSClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/ecs.py#L61' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesECSClient.run" class="hash-link"></a></Link></dt>
<dd>
Run ECS tasks, enriched with the pipes protocol.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>run_task_params</strong> (<em>dict</em>) – Parameters for the `run_task` boto3 ECS client call. Must contain `taskDefinition` key. See [Boto3 API Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html#run-task)
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional information to pass to the Pipes session in the external process.
- <strong>pipes_container_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – If running more than one container in the task, and using [`PipesCloudWatchMessageReader`](#dagster_aws.pipes.PipesCloudWatchMessageReader), specify the container name which will be running Pipes.
- <strong>waiter_config</strong> (<em>Optional</em><em>[</em><em>WaiterConfig</em><em>]</em>) – Optional waiter configuration to use. Defaults to 70 days (Delay: 6, MaxAttempts: 1000000).
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRClient'>`class` dagster_aws.pipes.PipesEMRClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr.py#L41' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for running jobs on AWS EMR.
Parameters:
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the EMR jobs. Recommended to use [`PipesS3MessageReader`](#dagster_aws.pipes.PipesS3MessageReader) with <cite>expect_s3_message_writer</cite> set to <cite>True</cite>.
- <strong>client</strong> (<em>Optional</em><em>[</em><em>boto3.client</em><em>]</em>) – The boto3 EMR client used to interact with AWS EMR.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into AWS EMR job. Defaults to `PipesEnvContextInjector`.
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the EMR job if the Dagster process receives a termination signal.
- <strong>wait_for_s3_logs_seconds</strong> (<em>int</em>) – The number of seconds to wait for S3 logs to be written after execution completes.
- <strong>s3_application_logs_prefix</strong> (<em>str</em>) – The prefix to use when looking for application logs in S3. Defaults to <cite>containers</cite>. Another common value is <cite>steps</cite> (for non-yarn clusters).
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRClient.run" class="hash-link"></a></Link></dt>
<dd>
Run a job on AWS EMR, enriched with the pipes protocol.
Starts a new EMR cluster for each invocation.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>run_job_flow_params</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) – Parameters for the `run_job_flow` boto3 EMR client call. See [Boto3 EMR API Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html)
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional information to pass to the Pipes session in the external process.
Returns: Wrapper containing results reported by the external process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRContainersClient'>`class` dagster_aws.pipes.PipesEMRContainersClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr_containers.py#L35' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRContainersClient" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A pipes client for running workloads on AWS EMR Containers.
Parameters:
- <strong>client</strong> (<em>Optional</em><em>[</em><em>boto3.client</em><em>]</em>) – The boto3 AWS EMR containers client used to interact with AWS EMR Containers.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into AWS EMR Containers workload. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the AWS EMR Containers workload. It’s recommended to use [`PipesS3MessageReader`](#dagster_aws.pipes.PipesS3MessageReader).
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the AWS EMR Containers workload if the Dagster process receives a termination signal.
- <strong>pipes_params_bootstrap_method</strong> (<em>Literal</em><em>[</em><em>"args"</em><em>, </em><em>"env"</em><em>]</em>) – The method to use to inject parameters into the AWS EMR Containers workload. Defaults to “args”.
- <strong>waiter_config</strong> (<em>Optional</em><em>[</em><em>WaiterConfig</em><em>]</em>) – Optional waiter configuration to use. Defaults to 70 days (Delay: 6, MaxAttempts: 1000000).
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRContainersClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr_containers.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRContainersClient.run" class="hash-link"></a></Link></dt>
<dd>
Run a workload on AWS EMR Containers, enriched with the pipes protocol.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>start_job_run_params</strong> (<em>dict</em>) – Parameters for the start_job_run boto3 AWS EMR Containers client call.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional information to pass to the Pipes session in the external process.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRServerlessClient'>`class` dagster_aws.pipes.PipesEMRServerlessClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr_serverless.py#L38' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRServerlessClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for running workloads on AWS EMR Serverless.
Parameters:
- <strong>client</strong> (<em>Optional</em><em>[</em><em>boto3.client</em><em>]</em>) – The boto3 AWS EMR Serverless client used to interact with AWS EMR Serverless.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into AWS EMR Serverless workload. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the AWS EMR Serverless workload. Defaults to [`PipesCloudWatchMessageReader`](#dagster_aws.pipes.PipesCloudWatchMessageReader).
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the AWS EMR Serverless workload if the Dagster process receives a termination signal.
- <strong>poll_interval</strong> (<em>float</em>) – The interval in seconds to poll the AWS EMR Serverless workload for status updates. Defaults to 5 seconds.
<dl>
<dt><Link class="anchor" id='dagster_aws.pipes.PipesEMRServerlessClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/pipes/clients/emr_serverless.py#L90' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.pipes.PipesEMRServerlessClient.run" class="hash-link"></a></Link></dt>
<dd>
Run a workload on AWS EMR Serverless, enriched with the pipes protocol.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>start_job_run_params</strong> (<em>dict</em>) – Parameters for the start_job_run boto3 AWS EMR Serverless client call.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional information to pass to the Pipes session in the external process.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
</div></div>
<div class="section" id="components">
## Components
Resource components for use with Dagster’s component system.
<div class="section" id="credentials-components">
### Credentials Components
<dl>
<dt><Link class="anchor" id='dagster_aws.Boto3CredentialsComponent'>`class` dagster_aws.Boto3CredentialsComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/credentials.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.Boto3CredentialsComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Configuration for standard AWS SDK (Boto3) credentials and session settings.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.S3CredentialsComponent'>`class` dagster_aws.S3CredentialsComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/credentials.py#L60' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.S3CredentialsComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Configuration for S3-specific credentials, including unsigned session support.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.AthenaCredentialsComponent'>`class` dagster_aws.AthenaCredentialsComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/credentials.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.AthenaCredentialsComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Configuration for Athena-specific authentication and query settings.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.RedshiftCredentialsComponent'>`class` dagster_aws.redshift.RedshiftCredentialsComponent<a href="#dagster_aws.redshift.RedshiftCredentialsComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Credentials and connection configuration for Redshift.
</dd>
</dl>
</div>
<div class="section" id="resource-components">
### Resource Components
<dl>
<dt><Link class="anchor" id='dagster_aws.S3ResourceComponent'>`class` dagster_aws.S3ResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/s3.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.S3ResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides an S3Resource for interacting with AWS S3.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.S3FileManagerResourceComponent'>`class` dagster_aws.S3FileManagerResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/s3.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.S3FileManagerResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides an S3FileManagerResource for file management operations on AWS S3.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.AthenaClientResourceComponent'>`class` dagster_aws.AthenaClientResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/athena.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.AthenaClientResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides an AthenaClientResource for executing queries against Amazon Athena.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.RedshiftClientResourceComponent'>`class` dagster_aws.redshift.RedshiftClientResourceComponent<a href="#dagster_aws.redshift.RedshiftClientResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a RedshiftClientResource for connecting to and querying Amazon Redshift.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.SSMResourceComponent'>`class` dagster_aws.SSMResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/ssm.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.SSMResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a SSMResource for interacting with AWS Systems Manager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.ParameterStoreResourceComponent'>`class` dagster_aws.ParameterStoreResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/ssm.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ParameterStoreResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a ParameterStoreResource for fetching parameters from AWS SSM Parameter Store.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.SecretsManagerResourceComponent'>`class` dagster_aws.SecretsManagerResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/secretsmanager.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.SecretsManagerResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a SecretsManagerResource for interacting with AWS Secrets Manager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.SecretsManagerSecretsResourceComponent'>`class` dagster_aws.SecretsManagerSecretsResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/secretsmanager.py#L38' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.SecretsManagerSecretsResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a SecretsManagerSecretsResource for fetching secrets from AWS Secrets Manager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.RDSResourceComponent'>`class` dagster_aws.RDSResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/rds.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.RDSResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides an RDSResource for interacting with AWS RDS service.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.ECRPublicResourceComponent'>`class` dagster_aws.ECRPublicResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/components/ecr.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.ECRPublicResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides an ECRPublicResource for connecting to AWS Public ECR.
</dd>
</dl>
</div></div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.ConfigurablePickledObjectS3IOManager'>dagster_aws.s3.ConfigurablePickledObjectS3IOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py#L159' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.ConfigurablePickledObjectS3IOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 2.0.
Please use S3PickleIOManager instead..
:::
Renamed to S3PickleIOManager. See S3PickleIOManager for documentation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.s3_resource'>dagster_aws.s3.s3_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/resources.py#L106' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.s3_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource that gives access to S3.
The underlying S3 session is created by calling
`boto3.session.Session(profile_name)`.
The returned resource object is an S3 client, an instance of <cite>botocore.client.S3</cite>.
Example:
```python
from dagster import build_op_context, job, op
from dagster_aws.s3 import s3_resource
@op(required_resource_keys={'s3'})
def example_s3_op(context):
return context.resources.s3.list_objects_v2(
Bucket='my-bucket',
Prefix='some-key'
)
@job(resource_defs={'s3': s3_resource})
def example_job():
example_s3_op()
example_job.execute_in_process(
run_config={
'resources': {
's3': {
'config': {
'region_name': 'us-west-1',
}
}
}
}
)
```
Note that your ops must also declare that they require this resource with
<cite>required_resource_keys</cite>, or it will not be initialized for the execution of their compute
functions.
You may configure this resource as follows:
```YAML
resources:
s3:
config:
region_name: "us-west-1"
# Optional[str]: Specifies a custom region for the S3 session. Default is chosen
# through the ordinary boto credential chain.
use_unsigned_session: false
# Optional[bool]: Specifies whether to use an unsigned S3 session. Default: True
endpoint_url: "http://localhost"
# Optional[str]: Specifies a custom endpoint for the S3 session. Default is None.
profile_name: "dev"
# Optional[str]: Specifies a custom profile for S3 session. Default is default
# profile as specified in ~/.aws/credentials file
use_ssl: true
# Optional[bool]: Whether or not to use SSL. By default, SSL is used.
verify: None
# Optional[str]: Whether or not to verify SSL certificates. By default SSL certificates are verified.
# You can also specify this argument if you want to use a different CA cert bundle than the one used by botocore."
aws_access_key_id: None
# Optional[str]: The access key to use when creating the client.
aws_secret_access_key: None
# Optional[str]: The secret key to use when creating the client.
aws_session_token: None
# Optional[str]: The session token to use when creating the client.
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.s3_pickle_io_manager'>dagster_aws.s3.s3_pickle_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/io_manager.py#L169' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.s3_pickle_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using S3 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for S3 and the backing bucket.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
1. Attach this IO manager to a set of assets.
```python
from dagster import Definitions, asset
from dagster_aws.s3 import s3_pickle_io_manager, s3_resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": s3_pickle_io_manager.configured(
{"s3_bucket": "my-cool-bucket", "s3_prefix": "my-cool-prefix"}
),
"s3": s3_resource,
},
)
```
2. Attach this IO manager to your job to make it available to your ops.
```python
from dagster import job
from dagster_aws.s3 import s3_pickle_io_manager, s3_resource
@job(
resource_defs={
"io_manager": s3_pickle_io_manager.configured(
{"s3_bucket": "my-cool-bucket", "s3_prefix": "my-cool-prefix"}
),
"s3": s3_resource,
},
)
def my_job():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.s3.s3_file_manager'>dagster_aws.s3.s3_file_manager ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/resources.py#L208' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.s3.s3_file_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
FileManager that provides abstract access to S3.
Implements the [`FileManager`](../../../api/dagster/internals.mdx#dagster._core.storage.file_manager.FileManager) API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.redshift_resource'>dagster_aws.redshift.redshift_resource ResourceDefinition<a href="#dagster_aws.redshift.redshift_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource enables connecting to a Redshift cluster and issuing queries against that
cluster.
Example:
```python
from dagster import build_op_context, op
from dagster_aws.redshift import redshift_resource
@op(required_resource_keys={'redshift'})
def example_redshift_op(context):
return context.resources.redshift.execute_query('SELECT 1', fetch_results=True)
redshift_configured = redshift_resource.configured({
'host': 'my-redshift-cluster.us-east-1.redshift.amazonaws.com',
'port': 5439,
'user': 'dagster',
'password': 'dagster',
'database': 'dev',
})
context = build_op_context(resources={'redshift': redshift_configured})
assert example_redshift_op(context) == [(1,)]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.redshift.fake_redshift_resource'>dagster_aws.redshift.fake_redshift_resource ResourceDefinition<a href="#dagster_aws.redshift.fake_redshift_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.secretsmanager.secretsmanager_resource'>dagster_aws.secretsmanager.secretsmanager_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/secretsmanager/resources.py#L79' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.secretsmanager.secretsmanager_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that gives access to AWS SecretsManager.
The underlying SecretsManager session is created by calling
`boto3.session.Session(profile_name)`.
The returned resource object is a SecretsManager client, an instance of <cite>botocore.client.SecretsManager</cite>.
Example:
```python
from dagster import build_op_context, job, op
from dagster_aws.secretsmanager import secretsmanager_resource
@op(required_resource_keys={'secretsmanager'})
def example_secretsmanager_op(context):
return context.resources.secretsmanager.get_secret_value(
SecretId='arn:aws:secretsmanager:region:aws_account_id:secret:appauthexample-AbCdEf'
)
@job(resource_defs={'secretsmanager': secretsmanager_resource})
def example_job():
example_secretsmanager_op()
example_job.execute_in_process(
run_config={
'resources': {
'secretsmanager': {
'config': {
'region_name': 'us-west-1',
}
}
}
}
)
```
You may configure this resource as follows:
```YAML
resources:
secretsmanager:
config:
region_name: "us-west-1"
# Optional[str]: Specifies a custom region for the SecretsManager session. Default is chosen
# through the ordinary boto credential chain.
profile_name: "dev"
# Optional[str]: Specifies a custom profile for SecretsManager session. Default is default
# profile as specified in ~/.aws/credentials file
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_aws.secretsmanager.secretsmanager_secrets_resource'>dagster_aws.secretsmanager.secretsmanager_secrets_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/secretsmanager/resources.py#L257' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_aws.secretsmanager.secretsmanager_secrets_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource that provides a dict which maps selected SecretsManager secrets to
their string values. Also optionally sets chosen secrets as environment variables.
Example:
```python
import os
from dagster import build_op_context, job, op
from dagster_aws.secretsmanager import secretsmanager_secrets_resource
@op(required_resource_keys={'secrets'})
def example_secretsmanager_secrets_op(context):
return context.resources.secrets.get("my-secret-name")
@op(required_resource_keys={'secrets'})
def example_secretsmanager_secrets_op_2(context):
return os.getenv("my-other-secret-name")
@job(resource_defs={'secrets': secretsmanager_secrets_resource})
def example_job():
example_secretsmanager_secrets_op()
example_secretsmanager_secrets_op_2()
example_job.execute_in_process(
run_config={
'resources': {
'secrets': {
'config': {
'region_name': 'us-west-1',
'secrets_tag': 'dagster',
'add_to_environment': True,
}
}
}
}
)
```
Note that your ops must also declare that they require this resource with
<cite>required_resource_keys</cite>, or it will not be initialized for the execution of their compute
functions.
You may configure this resource as follows:
```YAML
resources:
secretsmanager:
config:
region_name: "us-west-1"
# Optional[str]: Specifies a custom region for the SecretsManager session. Default is chosen
# through the ordinary boto credential chain.
profile_name: "dev"
# Optional[str]: Specifies a custom profile for SecretsManager session. Default is default
# profile as specified in ~/.aws/credentials file
secrets: ["arn:aws:secretsmanager:region:aws_account_id:secret:appauthexample-AbCdEf"]
# Optional[List[str]]: Specifies a list of secret ARNs to pull from SecretsManager.
secrets_tag: "dagster"
# Optional[str]: Specifies a tag, all secrets which have the tag set will be pulled
# from SecretsManager.
add_to_environment: true
# Optional[bool]: Whether to set the selected secrets as environment variables. Defaults
# to false.
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-azure library'
sidebar_position: 1000
title_meta: 'dagster-azure library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-azure library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-azure-library">
# dagster-azure library
Utilities for using Azure Storage Accounts with Dagster. This is mostly aimed at Azure Data Lake
Storage Gen 2 (ADLS2) but also contains some utilities for Azure Blob Storage.
<div class="section" id="components">
## Components
<dl>
<dt><Link class="anchor" id='dagster_azure.AzureBlobStorageResourceComponent'>`class` dagster_azure.AzureBlobStorageResourceComponent<a href="#dagster_azure.AzureBlobStorageResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Component for Azure Blob Storage Resource.
Wraps AzureBlobStorageResource for use in YAML-based configuration.
Example usage in YAML:
```YAML
resources:
- name: blob_storage
type: dagster_azure.AzureBlobStorageResourceComponent
attributes:
account_url: "https://myaccount.blob.core.windows.net"
credential:
credential_type: key
key: "{{ env.AZURE_STORAGE_ACCOUNT_KEY }}"
resource_key: blob_storage
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.ADLS2ResourceComponent'>`class` dagster_azure.ADLS2ResourceComponent<a href="#dagster_azure.ADLS2ResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Component for Azure Data Lake Storage Gen2 Resource.
Wraps ADLS2Resource for use in YAML-based configuration.
Example usage in YAML:
```YAML
resources:
- name: adls2
type: dagster_azure.ADLS2ResourceComponent
attributes:
storage_account: my_storage_account
credential:
credential_type: sas
token: "{{ env.ADLS2_SAS_TOKEN }}"
resource_key: adls2
```
</dd>
</dl>
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.ADLS2Resource'>dagster_azure.adls2.ADLS2Resource ResourceDefinition<a href="#dagster_azure.adls2.ADLS2Resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource containing clients to access Azure Data Lake Storage Gen2.
Contains a client for both the Data Lake and Blob APIs, to work around the limitations
of each.
Example usage:
Attach this resource to your Definitions to be used by assets and jobs.
```python
from dagster import Definitions, asset, job, op
from dagster_azure.adls2 import ADLS2Resource, ADLS2SASToken
@asset
def asset1(adls2: ADLS2Resource):
adls2.adls2_client.list_file_systems()
...
@op
def my_op(adls2: ADLS2Resource):
adls2.adls2_client.list_file_systems()
...
@job
def my_job():
my_op()
Definitions(
assets=[asset1],
jobs=[my_job],
resources={
"adls2": ADLS2Resource(
storage_account="my-storage-account",
credential=ADLS2SASToken(token="my-sas-token"),
)
},
)
```
Attach this resource to your job to make it available to your ops.
```python
from dagster import job, op
from dagster_azure.adls2 import ADLS2Resource, ADLS2SASToken
@op
def my_op(adls2: ADLS2Resource):
adls2.adls2_client.list_file_systems()
...
@job(
resource_defs={
"adls2": ADLS2Resource(
storage_account="my-storage-account",
credential=ADLS2SASToken(token="my-sas-token"),
)
},
)
def my_job():
my_op()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.fakes.FakeADLS2Resource'>dagster_azure.fakes.FakeADLS2Resource ResourceDefinition<a href="#dagster_azure.fakes.FakeADLS2Resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Stateful mock of an ADLS2Resource for testing.
Wraps a `mock.MagicMock`. Containers are implemented using an in-memory dict.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.blob.AzureBlobStorageResource'>dagster_azure.blob.AzureBlobStorageResource ResourceDefinition<a href="#dagster_azure.blob.AzureBlobStorageResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with Azure Blob Storage.
Examples:
```python
import os
from dagster import Definitions, asset, EnvVar
from dagster_azure.blob import (
AzureBlobStorageResource,
AzureBlobStorageKeyCredential,
AzureBlobStorageDefaultCredential
)
@asset
def my_table(azure_blob_storage: AzureBlobStorageResource):
with azure_blob_storage.get_client() as blob_storage_client:
response = blob_storage_client.list_containers()
Definitions(
assets=[my_table],
resources={
"azure_blob_storage": AzureBlobStorageResource(
account_url=EnvVar("AZURE_BLOB_STORAGE_ACCOUNT_URL"),
credential=AzureBlobStorageDefaultCredential() if os.getenv("DEV") else
AzureBlobStorageKeyCredential(key=EnvVar("AZURE_BLOB_STORAGE_KEY"))
),
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.blob.AzureBlobComputeLogManager'>`class` dagster_azure.blob.AzureBlobComputeLogManager<a href="#dagster_azure.blob.AzureBlobComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
Logs op compute function stdout and stderr to Azure Blob Storage.
This is also compatible with Azure Data Lake Storage.
Users should not instantiate this class directly. Instead, use a YAML block in `dagster.yaml`. Examples provided below
will show how to configure with various credentialing schemes.
Parameters:
- <strong>storage_account</strong> (<em>str</em>) – The storage account name to which to log.
- <strong>container</strong> (<em>str</em>) – The container (or ADLS2 filesystem) to which to log.
- <strong>secret_credential</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) – Secret credential for the storage account. This should be a dictionary with keys <cite>client_id</cite>, <cite>client_secret</cite>, and <cite>tenant_id</cite>.
- <strong>access_key_or_sas_token</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Access key or SAS token for the storage account.
- <strong>default_azure_credential</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) – Use and configure DefaultAzureCredential. Cannot be used with sas token or secret key config.
- <strong>local_dir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Path to the local directory in which to stage logs. Default: `dagster_shared.seven.get_system_temp_directory()`.
- <strong>prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Prefix for the log file keys.
- <strong>upload_interval</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – Interval in seconds to upload partial log files blob storage. By default, will only upload when the capture is complete.
- <strong>show_url_only</strong> (<em>bool</em>) – Only show the URL of the log file in the UI, instead of fetching and displaying the full content. Default False.
- <strong>inst_data</strong> (<em>Optional</em><em>[</em>[*ConfigurableClassData*](../../../api/dagster/internals.mdx#dagster._serdes.ConfigurableClassData)<em>]</em>) – Serializable representation of the compute log manager when newed up from config.
Examples:
Using an Azure Blob Storage account with an [AzureSecretCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.clientsecretcredential?view=azure-python):
```YAML
compute_logs:
module: dagster_azure.blob.compute_log_manager
class: AzureBlobComputeLogManager
config:
storage_account: my-storage-account
container: my-container
secret_credential:
client_id: my-client-id
client_secret: my-client-secret
tenant_id: my-tenant-id
prefix: "dagster-test-"
local_dir: "/tmp/cool"
upload_interval: 30
show_url_only: false
```
Using an Azure Blob Storage account with a [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python):
```YAML
compute_logs:
module: dagster_azure.blob.compute_log_manager
class: AzureBlobComputeLogManager
config:
storage_account: my-storage-account
container: my-container
default_azure_credential:
exclude_environment_credential: false
prefix: "dagster-test-"
local_dir: "/tmp/cool"
upload_interval: 30
show_url_only: false
```
Using an Azure Blob Storage account with an access key:
```YAML
compute_logs:
module: dagster_azure.blob.compute_log_manager
class: AzureBlobComputeLogManager
config:
storage_account: my-storage-account
container: my-container
access_key_or_sas_token: my-access-key
prefix: "dagster-test-"
local_dir: "/tmp/cool"
upload_interval: 30
show_url_only: false
```
</dd>
</dl>
</div>
<div class="section" id="i-o-manager">
## I/O manager
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.ADLS2PickleIOManager'>dagster_azure.adls2.ADLS2PickleIOManager IOManagerDefinition<a href="#dagster_azure.adls2.ADLS2PickleIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using Azure Data Lake Storage Gen2 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for ADLS and the backing
container.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
1. Attach this IO manager to a set of assets.
```python
from dagster import Definitions, asset
from dagster_azure.adls2 import ADLS2PickleIOManager, ADLS2Resource, ADLS2SASToken
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return df[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": ADLS2PickleIOManager(
adls2_file_system="my-cool-fs",
adls2_prefix="my-cool-prefix",
adls2=ADLS2Resource(
storage_account="my-storage-account",
credential=ADLS2SASToken(token="my-sas-token"),
),
),
},
)
```
2. Attach this IO manager to your job to make it available to your ops.
```python
from dagster import job
from dagster_azure.adls2 import ADLS2PickleIOManager, ADLS2Resource, ADLS2SASToken
@job(
resource_defs={
"io_manager": ADLS2PickleIOManager(
adls2_file_system="my-cool-fs",
adls2_prefix="my-cool-prefix",
adls2=ADLS2Resource(
storage_account="my-storage-account",
credential=ADLS2SASToken(token="my-sas-token"),
),
),
},
)
def my_job():
...
```
</dd>
</dl>
</div>
<div class="section" id="file-manager">
## File manager
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.adls2_file_manager'>dagster_azure.adls2.adls2_file_manager ResourceDefinition<a href="#dagster_azure.adls2.adls2_file_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
FileManager that provides abstract access to ADLS2.
Implements the [`FileManager`](../../../api/dagster/internals.mdx#dagster._core.storage.file_manager.FileManager) API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.ADLS2FileHandle'>`class` dagster_azure.adls2.ADLS2FileHandle<a href="#dagster_azure.adls2.ADLS2FileHandle" class="hash-link"></a></Link></dt>
<dd>
A reference to a file on ADLS2.
</dd>
</dl>
</div>
<div class="section" id="pipes">
## Pipes
<dl>
<dt><Link class="anchor" id='dagster_azure.pipes.PipesAzureBlobStorageContextInjector'>`class` dagster_azure.pipes.PipesAzureBlobStorageContextInjector<a href="#dagster_azure.pipes.PipesAzureBlobStorageContextInjector" class="hash-link"></a></Link></dt>
<dd>
A context injector that injects context by writing to a temporary AzureBlobStorage location.
Parameters:
- <strong>container</strong> (<em>str</em>) – The AzureBlobStorage container to write to.
- <strong>client</strong> (<em>azure.storage.blob.BlobServiceClient</em>) – An Azure Blob Storage client.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the Azure Blob Storage key. Defaults to a random string.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.pipes.PipesAzureBlobStorageMessageReader'>`class` dagster_azure.pipes.PipesAzureBlobStorageMessageReader<a href="#dagster_azure.pipes.PipesAzureBlobStorageMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by periodically reading message chunks from a specified AzureBlobStorage
container.
If <cite>log_readers</cite> is passed, this reader will also start the passed readers
when the first message is received from the external process.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a chunk
- <strong>container</strong> (<em>str</em>) – The AzureBlobStorage container to read from.
- <strong>client</strong> (<em>azure.storage.blob.BlobServiceClient</em>) – An azure BlobServiceClient.
- <strong>log_readers</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>PipesLogReader</em><em>]</em><em>]</em>) – A set of log readers for logs on AzureBlobStorage.
- <strong>include_stdio_in_messages</strong> (<em>bool</em>) – Whether to send stdout/stderr to Dagster via Pipes messages. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.pipes.clients.PipesAzureMLClient'>`class` dagster_azure.pipes.clients.PipesAzureMLClient<a href="#dagster_azure.pipes.clients.PipesAzureMLClient" class="hash-link"></a></Link></dt>
<dd>
Pipes client for Azure ML.
Parameters:
- <strong>client</strong> (<em>MLClient</em>) – An Azure ML <cite>MLClient</cite> object.
- <strong>context_injector</strong> ([*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)) – A context injector to use to inject context into the Azure ML job process.
- <strong>message_reader</strong> ([*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)) – A message reader to use to read messages from the Azure ML job.
- <strong>poll_interval_seconds</strong> (<em>float</em>) – How long to sleep between checking the status of the job run. Defaults to 5.
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the Azure ML job if the orchestration process is interrupted or canceled. Defaults to True.
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.ConfigurablePickledObjectADLS2IOManager'>dagster_azure.adls2.ConfigurablePickledObjectADLS2IOManager IOManagerDefinition<a href="#dagster_azure.adls2.ConfigurablePickledObjectADLS2IOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 2.0.
Please use ADLS2PickleIOManager instead..
:::
Renamed to ADLS2PickleIOManager. See ADLS2PickleIOManager for documentation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.adls2_resource'>dagster_azure.adls2.adls2_resource ResourceDefinition<a href="#dagster_azure.adls2.adls2_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource that gives ops access to Azure Data Lake Storage Gen2.
The underlying client is a `DataLakeServiceClient`.
Attach this resource definition to a [`JobDefinition`](../../../api/dagster/jobs.mdx#dagster.JobDefinition) in order to make it
available to your ops.
Example:
```python
from dagster import job, op
from dagster_azure.adls2 import adls2_resource
@op(required_resource_keys={'adls2'})
def example_adls2_op(context):
return list(context.resources.adls2.adls2_client.list_file_systems())
@job(resource_defs={"adls2": adls2_resource})
def my_job():
example_adls2_op()
```
Note that your ops must also declare that they require this resource with
<cite>required_resource_keys</cite>, or it will not be initialized for the execution of their compute
functions.
You may pass credentials to this resource using either a SAS token, a key or by passing the
<cite>DefaultAzureCredential</cite> object.
```YAML
resources:
adls2:
config:
storage_account: my_storage_account
# str: The storage account name.
credential:
sas: my_sas_token
# str: the SAS token for the account.
key:
env: AZURE_DATA_LAKE_STORAGE_KEY
# str: The shared access key for the account.
DefaultAzureCredential: {}
# dict: The keyword arguments used for DefaultAzureCredential
# or leave the object empty for no arguments
DefaultAzureCredential:
exclude_environment_credential: true
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_azure.adls2.adls2_pickle_io_manager'>dagster_azure.adls2.adls2_pickle_io_manager IOManagerDefinition<a href="#dagster_azure.adls2.adls2_pickle_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using Azure Data Lake Storage Gen2 for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for ADLS and the backing
container.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at “\<base_dir>/\<asset_key>”. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of “/my/base/path”, an asset with key
<cite>AssetKey([“one”, “two”, “three”])</cite> would be stored in a file called “three” in a directory
with path “/my/base/path/one/two/”.
Example usage:
Attach this IO manager to a set of assets.
```python
from dagster import Definitions, asset
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return df[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": adls2_pickle_io_manager.configured(
{"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
),
"adls2": adls2_resource,
},
)
```
Attach this IO manager to your job to make it available to your ops.
```python
from dagster import job
from dagster_azure.adls2 import adls2_pickle_io_manager, adls2_resource
@job(
resource_defs={
"io_manager": adls2_pickle_io_manager.configured(
{"adls2_file_system": "my-cool-fs", "adls2_prefix": "my-cool-prefix"}
),
"adls2": adls2_resource,
},
)
def my_job():
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-celery-docker library'
sidebar_position: 1000
title_meta: 'dagster-celery-docker library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-celery-docker library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-celery-docker-library">
# dagster-celery-docker library
<div class="section" id="apis">
## APIs
<dl>
<dt><Link class="anchor" id='dagster_celery_docker.celery_docker_executor'>dagster_celery_docker.celery_docker_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery-docker/dagster_celery_docker/executor.py#L77' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_celery_docker.celery_docker_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Celery-based executor which launches tasks in docker containers.
The Celery executor exposes config settings for the underlying Celery app under
the `config_source` key. This config corresponds to the “new lowercase settings” introduced
in Celery version 4.0 and the object constructed from config will be passed to the
`celery.Celery` constructor as its `config_source` argument.
(See [https://docs.celeryq.dev/en/stable/userguide/configuration.html](https://docs.celeryq.dev/en/stable/userguide/configuration.html) for details.)
The executor also exposes the `broker`, <cite>backend</cite>, and `include` arguments to the
`celery.Celery` constructor.
In the most common case, you may want to modify the `broker` and `backend` (e.g., to use
Redis instead of RabbitMQ). We expect that `config_source` will be less frequently
modified, but that when op executions are especially fast or slow, or when there are
different requirements around idempotence or retry, it may make sense to execute jobs
with variations on these settings.
To use the <cite>celery_docker_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster import job
from dagster_celery_docker.executor import celery_docker_executor
@job(executor_def=celery_docker_executor)
def celery_enabled_job():
pass
```
Then you can configure the executor as follows:
```YAML
execution:
config:
docker:
image: 'my_repo.com/image_name:latest'
registry:
url: 'my_repo.com'
username: 'my_user'
password: {env: 'DOCKER_PASSWORD'}
env_vars: ["DAGSTER_HOME"] # environment vars to pass from celery worker to docker
container_kwargs: # keyword args to be passed to the container. example:
volumes: ['/home/user1/:/mnt/vol2','/var/www:/mnt/vol1']
broker: 'pyamqp://guest@localhost//' # Optional[str]: The URL of the Celery broker
backend: 'rpc://' # Optional[str]: The URL of the Celery results backend
include: ['my_module'] # Optional[List[str]]: Modules every worker should import
config_source: # Dict[str, Any]: Any additional parameters to pass to the
#... # Celery workers. This dict will be passed as the `config_source`
#... # argument of celery.Celery().
```
Note that the YAML you provide here must align with the configuration with which the Celery
workers on which you hope to run were started. If, for example, you point the executor at a
different broker than the one your workers are listening to, the workers will never be able to
pick up tasks for execution.
In deployments where the celery_docker_job_executor is used all appropriate celery and dagster_celery
commands must be invoked with the <cite>-A dagster_celery_docker.app</cite> argument.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-celery-k8s library'
sidebar_position: 1000
title_meta: 'dagster-celery-k8s library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-celery-k8s library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-celery-k8s-library">
# dagster-celery-k8s library
<div class="section" id="apis">
## APIs
<dl>
<dt><Link class="anchor" id='dagster_celery_k8s.CeleryK8sRunLauncher'>dagster_celery_k8s.CeleryK8sRunLauncher RunLauncher<a href="#dagster_celery_k8s.CeleryK8sRunLauncher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
In contrast to the `K8sRunLauncher`, which launches dagster runs as single K8s
Jobs, this run launcher is intended for use in concert with
[`dagster_celery_k8s.celery_k8s_job_executor()`](#dagster_celery_k8s.celery_k8s_job_executor).
With this run launcher, execution is delegated to:
>
1. A run worker Kubernetes Job, which traverses the dagster run execution plan and submits steps to Celery queues for execution;
2. The step executions which are submitted to Celery queues are picked up by Celery workers, and each step execution spawns a step execution Kubernetes Job. See the implementation defined in `dagster_celery_k8.executor.create_k8s_job_task()`.
You can configure a Dagster instance to use this RunLauncher by adding a section to your
`dagster.yaml` like the following:
```yaml
run_launcher:
module: dagster_k8s.launcher
class: CeleryK8sRunLauncher
config:
instance_config_map: "dagster-k8s-instance-config-map"
dagster_home: "/some/path"
postgres_password_secret: "dagster-k8s-pg-password"
broker: "some_celery_broker_url"
backend: "some_celery_backend_url"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_celery_k8s.celery_k8s_job_executor'>dagster_celery_k8s.celery_k8s_job_executor ExecutorDefinition<a href="#dagster_celery_k8s.celery_k8s_job_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Celery-based executor which launches tasks as Kubernetes Jobs.
The Celery executor exposes config settings for the underlying Celery app under
the `config_source` key. This config corresponds to the “new lowercase settings” introduced
in Celery version 4.0 and the object constructed from config will be passed to the
`celery.Celery` constructor as its `config_source` argument.
(See [https://docs.celeryq.dev/en/stable/userguide/configuration.html](https://docs.celeryq.dev/en/stable/userguide/configuration.html) for details.)
The executor also exposes the `broker`, <cite>backend</cite>, and `include` arguments to the
`celery.Celery` constructor.
In the most common case, you may want to modify the `broker` and `backend` (e.g., to use
Redis instead of RabbitMQ). We expect that `config_source` will be less frequently
modified, but that when op executions are especially fast or slow, or when there are
different requirements around idempotence or retry, it may make sense to execute dagster jobs
with variations on these settings.
To use the <cite>celery_k8s_job_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster import job
from dagster_celery_k8s.executor import celery_k8s_job_executor
@job(executor_def=celery_k8s_job_executor)
def celery_enabled_job():
pass
```
Then you can configure the executor as follows:
```YAML
execution:
config:
job_image: 'my_repo.com/image_name:latest'
job_namespace: 'some-namespace'
broker: 'pyamqp://guest@localhost//' # Optional[str]: The URL of the Celery broker
backend: 'rpc://' # Optional[str]: The URL of the Celery results backend
include: ['my_module'] # Optional[List[str]]: Modules every worker should import
config_source: # Dict[str, Any]: Any additional parameters to pass to the
#... # Celery workers. This dict will be passed as the `config_source`
#... # argument of celery.Celery().
```
Note that the YAML you provide here must align with the configuration with which the Celery
workers on which you hope to run were started. If, for example, you point the executor at a
different broker than the one your workers are listening to, the workers will never be able to
pick up tasks for execution.
In deployments where the celery_k8s_job_executor is used all appropriate celery and dagster_celery
commands must be invoked with the <cite>-A dagster_celery_k8s.app</cite> argument.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-celery library'
sidebar_position: 1000
title_meta: 'dagster-celery library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-celery library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-celery-library">
# dagster-celery library
<div class="section" id="quickstart">
## Quickstart
To get a local rabbitmq broker started and available via the default
`pyamqp://guest@localhost:5672`, in the `dagster/python_modules/libraries/dagster-celery/`
directory run:
```bash
docker compose up
```
To run a celery worker:
```bash
celery -A dagster_celery.app worker -l info
```
To start multiple workers in the background, run:
```bash
celery multi start w2 -A dagster_celery.app -l info
```
To execute a job using the celery-backed executor, you’ll need to set the job’s `executor_def` to
the celery_executor.
```python
from dagster import job
from dagster_celery import celery_executor
@job(executor_def=celery_executor)
def my_job():
pass
```
<div class="section" id="monitoring-your-celery-tasks">
### Monitoring your Celery tasks
We advise using [Flower](https://celery.readthedocs.io/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor):
```bash
celery -A dagster_celery.app flower
```
</div>
<div class="section" id="customizing-the-celery-broker-backend-and-other-app-configuration">
### Customizing the Celery broker, backend, and other app configuration
By default this will use `amqp://guest:**@localhost:5672//` as the Celery broker URL and
`rpc://` as the results backend. In production, you will want to change these values. Pending the
introduction of a dagster_celery CLI, that would entail writing a Python module `my_module` as
follows:
```python
from celery import Celery
from dagster_celery.tasks import create_task
app = Celery('dagster', broker_url='some://custom@value', ...)
execute_plan = create_task(app)
if __name__ == '__main__':
app.worker_main()
```
You can then run the celery worker using:
```bash
celery -A my_module worker --loglevel=info
```
This customization mechanism is used to implement <cite>dagster_celery_k8s</cite> and <cite>dagster_celery_k8s</cite> which delegate the execution of steps to ephemeral kubernetes pods and docker containers, respectively.
</div></div>
<div class="section" id="api">
## API
<dl>
<dt><Link class="anchor" id='dagster_celery.celery_executor'>dagster_celery.celery_executor ExecutorDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery/dagster_celery/executor.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_celery.celery_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Celery-based executor.
The Celery executor exposes config settings for the underlying Celery app under
the `config_source` key. This config corresponds to the “new lowercase settings” introduced
in Celery version 4.0 and the object constructed from config will be passed to the
`celery.Celery` constructor as its `config_source` argument.
(See [https://docs.celeryq.dev/en/stable/userguide/configuration.html](https://docs.celeryq.dev/en/stable/userguide/configuration.html) for details.)
The executor also exposes the `broker`, <cite>backend</cite>, and `include` arguments to the
`celery.Celery` constructor.
In the most common case, you may want to modify the `broker` and `backend` (e.g., to use
Redis instead of RabbitMQ). We expect that `config_source` will be less frequently
modified, but that when solid executions are especially fast or slow, or when there are
different requirements around idempotence or retry, it may make sense to execute jobs
with variations on these settings.
To use the <cite>celery_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster import job
from dagster_celery import celery_executor
@job(executor_def=celery_executor)
def celery_enabled_job():
pass
```
Then you can configure the executor as follows:
```YAML
execution:
config:
broker: 'pyamqp://guest@localhost//' # Optional[str]: The URL of the Celery broker
backend: 'rpc://' # Optional[str]: The URL of the Celery results backend
include: ['my_module'] # Optional[List[str]]: Modules every worker should import
config_source: # Dict[str, Any]: Any additional parameters to pass to the
#... # Celery workers. This dict will be passed as the `config_source`
#... # argument of celery.Celery().
```
Note that the YAML you provide here must align with the configuration with which the Celery
workers on which you hope to run were started. If, for example, you point the executor at a
different broker than the one your workers are listening to, the workers will never be able to
pick up tasks for execution.
</dd>
</dl>
</div>
<div class="section" id="cli">
## CLI
The `dagster-celery` CLI lets you start, monitor, and terminate workers.
<div class="section" id="dagster-celery-worker-start">
### dagster-celery worker start
Start a dagster celery worker.
```shell
dagster-celery worker start [OPTIONS] [ADDITIONAL_ARGS]...
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-n'>-n, --name \<name><a href="#cmdoption-dagster-celery-worker-start-n" class="hash-link"></a></Link></dt>
<dd>
The name of the worker. Defaults to a unique name prefixed with “dagster-” and ending with the hostname.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-y'>-y, --config-yaml \<config_yaml><a href="#cmdoption-dagster-celery-worker-start-y" class="hash-link"></a></Link></dt>
<dd>
Specify the path to a config YAML file with options for the worker. This is the same config block that you provide to dagster_celery.celery_executor when configuring a job for execution with Celery, with, e.g., the URL of the broker to use.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-q'>-q, --queue \<queue><a href="#cmdoption-dagster-celery-worker-start-q" class="hash-link"></a></Link></dt>
<dd>
Names of the queues on which this worker should listen for tasks. Provide multiple -q arguments to specify multiple queues. Note that each celery worker may listen on no more than four queues.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-d'>-d, --background<a href="#cmdoption-dagster-celery-worker-start-d" class="hash-link"></a></Link></dt>
<dd>
Set this flag to run the worker in the background.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-i'>-i, --includes \<includes><a href="#cmdoption-dagster-celery-worker-start-i" class="hash-link"></a></Link></dt>
<dd>
Python modules the worker should import. Provide multiple -i arguments to specify multiple modules.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-l'>-l, --loglevel \<loglevel><a href="#cmdoption-dagster-celery-worker-start-l" class="hash-link"></a></Link></dt>
<dd>
Log level for the worker.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-A'>-A, --app \<app><a href="#cmdoption-dagster-celery-worker-start-A" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-start-arg-ADDITIONAL_ARGS'>ADDITIONAL_ARGS<a href="#cmdoption-dagster-celery-worker-start-arg-ADDITIONAL_ARGS" class="hash-link"></a></Link></dt>
<dd>
Optional argument(s)
</dd>
</dl>
</div>
<div class="section" id="dagster-celery-worker-list">
### dagster-celery worker list
List running dagster-celery workers. Note that we use the broker to contact the workers.
```shell
dagster-celery worker list [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-list-y'>-y, --config-yaml \<config_yaml><a href="#cmdoption-dagster-celery-worker-list-y" class="hash-link"></a></Link></dt>
<dd>
Specify the path to a config YAML file with options for the workers you are trying to manage. This is the same config block that you provide to dagster_celery.celery_executor when configuring a job for execution with Celery, with, e.g., the URL of the broker to use. Without this config file, you will not be able to find your workers (since the CLI won’t know how to reach the broker).
</dd>
</dl>
</div>
<div class="section" id="dagster-celery-worker-terminate">
### dagster-celery worker terminate
Shut down dagster-celery workers. Note that we use the broker to send signals to the workers to terminate – if the broker is not running, this command is a no-op. Provide the argument NAME to terminate a specific worker by name.
```shell
dagster-celery worker terminate [OPTIONS] [NAME]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-terminate-a'>-a, --all<a href="#cmdoption-dagster-celery-worker-terminate-a" class="hash-link"></a></Link></dt>
<dd>
Set this flag to terminate all running workers.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-terminate-y'>-y, --config-yaml \<config_yaml><a href="#cmdoption-dagster-celery-worker-terminate-y" class="hash-link"></a></Link></dt>
<dd>
Specify the path to a config YAML file with options for the workers you are trying to manage. This is the same config block that you provide to dagster_celery.celery_executor when configuring a job for execution with Celery, with, e.g., the URL of the broker to use. Without this config file, you will not be able to terminate your workers (since the CLI won’t know how to reach the broker).
</dd>
</dl>
Arguments:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-celery-worker-terminate-arg-NAME'>NAME<a href="#cmdoption-dagster-celery-worker-terminate-arg-NAME" class="hash-link"></a></Link></dt>
<dd>
Optional argument
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-census library'
sidebar_position: 1000
title_meta: 'dagster-census library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-census library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-census-library">
# dagster-census library
This library provides an integration with Census.
<dl>
<dt><Link class="anchor" id='dagster_census.CensusComponent'>`class` dagster_census.CensusComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/components/census_component.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.CensusComponent" class="hash-link"></a></Link></dt>
<dd>
Loads Census syncs from a Census workspace as Dagster assets.
Materializing these assets will trigger the Census sync, enabling
you to schedule Census syncs using Dagster.
Example:
```yaml
# defs.yaml
type: dagster_census.CensusComponent
attributes:
workspace:
api_key: "{{ env.CENSUS_API_KEY }}"
sync_selector:
by_name:
- my_first_sync
- my_second_sync
```
<dl>
<dt><Link class="anchor" id='dagster_census.CensusComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/components/census_component.py#L154' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.CensusComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes a Census sync for the selected sync.
This method can be overridden in a subclass to customize the sync execution behavior,
such as adding custom logging or handling sync results differently.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>census</strong> – The CensusResource used to trigger and monitor syncs
Returns: MaterializeResult event from the Census sync
Example:
Override this method to add custom logging during sync execution:
```python
from dagster_census import CensusComponent
import dagster as dg
class CustomCensusComponent(CensusComponent):
def execute(self, context, census):
context.log.info(f"Starting Census sync for {context.asset_key}")
result = super().execute(context, census)
context.log.info("Census sync completed successfully")
return result
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_census.CensusComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/components/census_component.py#L131' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.CensusComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Returns the `AssetSpec` representing a given Census sync.
</dd>
</dl>
</dd>
</dl>
To use the Census component, see the [Census component integration guide](https://docs.dagster.io/integrations/libraries/census).
<dl>
<dt><Link class="anchor" id='dagster_census.census_trigger_sync_op'>dagster_census.census_trigger_sync_op OpDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/ops.py#L9' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.census_trigger_sync_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Executes a Census sync for a given `sync_id` and polls until that sync completes, raising
an error if it is unsuccessful.
It outputs a [`CensusOutput`](#dagster_census.CensusOutput) which contains the details of the Census
sync after it successfully completes.
It requires the use of the `census_resource`, which allows it to
communicate with the Census API.
<strong>Examples:</strong>
```python
from dagster import job
from dagster_census import census_resource, census_sync_op
my_census_resource = census_resource.configured(
{
"api_key": {"env": "CENSUS_API_KEY"},
}
)
sync_foobar = census_sync_op.configured({"sync_id": "foobar"}, name="sync_foobar")
@job(resource_defs={"census": my_census_resource})
def my_simple_census_job():
sync_foobar()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_census.CensusResource'>`class` dagster_census.CensusResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/resources.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.CensusResource" class="hash-link"></a></Link></dt>
<dd>
This resource allows users to programatically interface with the Census REST API to launch
syncs and monitor their progress. This currently implements only a subset of the functionality
exposed by the API.
<strong>Examples:</strong>
```python
import dagster as dg
from dagster_census import CensusResource
census_resource = CensusResource(
api_key=dg.EnvVar("CENSUS_API_KEY")
)
@dg.asset
def census_sync_asset(census: CensusResource):
census.trigger_sync_and_poll(sync_id=123456)
defs = dg.Definitions(
assets=[census_sync_asset],
resources={"census": census_resource}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_census.CensusOutput'>`class` dagster_census.CensusOutput <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-census/dagster_census/types.py#L5' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_census.CensusOutput" class="hash-link"></a></Link></dt>
<dd>
Contains recorded information about the state of a Census sync after a sync completes.
Parameters:
- <strong>sync_run</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The details of the specific sync run.
- <strong>source</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – Information about the source for the Census sync.
- <strong>destination</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – Information about the destination for the Census sync.
</dd>
</dl>
</div>
---
---
title: 'dagster-clickhouse-pandas library'
sidebar_position: 1000
title_meta: 'dagster-clickhouse-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-clickhouse-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-clickhouse-pandas-library">
# dagster-clickhouse-pandas library
This library provides an integration with [ClickHouse](https://clickhouse.com) for [Pandas](https://pandas.pydata.org/) DataFrames.
Related guides:
- [Using Dagster with ClickHouse](https://docs.dagster.io/integrations/libraries/clickhouse)
- [ClickHouse integration reference](https://docs.dagster.io/integrations/libraries/clickhouse/reference)
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_pandas.ClickhousePandasIOManager'>dagster_clickhouse_pandas.ClickhousePandasIOManager IOManagerDefinition<a href="#dagster_clickhouse_pandas.ClickhousePandasIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
I/O manager for Pandas DataFrames stored in ClickHouse.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_pandas.ClickhousePandasTypeHandler'>`class` dagster_clickhouse_pandas.ClickhousePandasTypeHandler<a href="#dagster_clickhouse_pandas.ClickhousePandasTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Stores and loads Pandas DataFrames in ClickHouse via `clickhouse-driver`.
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_pandas.clickhouse_pandas_io_manager'>dagster_clickhouse_pandas.clickhouse_pandas_io_manager IOManagerDefinition<a href="#dagster_clickhouse_pandas.clickhouse_pandas_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager that reads and writes Pandas DataFrames to ClickHouse. Unannotated inputs and outputs
default to `pandas.DataFrame`.
Returns: IOManagerDefinition
</dd>
</dl>
</div></div>
---
---
title: 'dagster-clickhouse-polars library'
sidebar_position: 1000
title_meta: 'dagster-clickhouse-polars library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-clickhouse-polars library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-clickhouse-polars-library">
# dagster-clickhouse-polars library
This library provides an integration with [ClickHouse](https://clickhouse.com) for [Polars](https://pola.rs/) DataFrames.
Related guides:
- [Using Dagster with ClickHouse](https://docs.dagster.io/integrations/libraries/clickhouse)
- [ClickHouse integration reference](https://docs.dagster.io/integrations/libraries/clickhouse/reference)
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_polars.ClickhousePolarsIOManager'>dagster_clickhouse_polars.ClickhousePolarsIOManager IOManagerDefinition<a href="#dagster_clickhouse_polars.ClickhousePolarsIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
I/O manager for Polars DataFrames stored in ClickHouse.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_polars.ClickhousePolarsTypeHandler'>`class` dagster_clickhouse_polars.ClickhousePolarsTypeHandler<a href="#dagster_clickhouse_polars.ClickhousePolarsTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Stores and loads Polars DataFrames in ClickHouse via `clickhouse-driver`.
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_clickhouse_polars.clickhouse_polars_io_manager'>dagster_clickhouse_polars.clickhouse_polars_io_manager IOManagerDefinition<a href="#dagster_clickhouse_polars.clickhouse_polars_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager that reads and writes Polars DataFrames to ClickHouse. Unannotated inputs and outputs
default to `polars.DataFrame`.
Returns: IOManagerDefinition
</dd>
</dl>
</div></div>
---
---
title: 'dagster-clickhouse library'
sidebar_position: 1000
title_meta: 'dagster-clickhouse library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-clickhouse library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-clickhouse-library">
# dagster-clickhouse library
This library provides an integration with [ClickHouse](https://clickhouse.com) using the native
protocol via [clickhouse-driver](https://clickhouse-driver.readthedocs.io/).
Related guides:
- [Using Dagster with ClickHouse](https://docs.dagster.io/integrations/libraries/clickhouse)
- [ClickHouse integration reference](https://docs.dagster.io/integrations/libraries/clickhouse/reference)
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.ClickhouseResource'>dagster_clickhouse.ClickhouseResource ResourceDefinition<a href="#dagster_clickhouse.ClickhouseResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with a ClickHouse database via `clickhouse-driver` (native protocol).
Examples:
```python
from dagster import Definitions, asset
from dagster_clickhouse import ClickhouseResource
@asset
def my_table(clickhouse: ClickhouseResource):
with clickhouse.get_connection() as client:
client.execute("SELECT * FROM my_database.my_table LIMIT 1")
Definitions(
assets=[my_table],
resources={"clickhouse": ClickhouseResource(host="localhost")},
)
```
</dd>
</dl>
</div>
<div class="section" id="i-o-manager">
## I/O manager
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.ClickhouseIOManager'>dagster_clickhouse.ClickhouseIOManager IOManagerDefinition<a href="#dagster_clickhouse.ClickhouseIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager that reads inputs from and writes outputs to ClickHouse.
Asset and op metadata `schema` (and the I/O manager `schema` config) refer to the ClickHouse
<strong>database</strong> that contains the table.
The `database` field is the default database <strong>on the ClickHouse connection</strong> (server-side),
not the database that qualifies table names in Dagster metadata (that is `schema`).
Examples:
Subclass and implement `type_handlers()` with handlers from `dagster-clickhouse-pandas`
or `dagster-clickhouse-polars`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.build_clickhouse_io_manager'>dagster_clickhouse.build_clickhouse_io_manager<a href="#dagster_clickhouse.build_clickhouse_io_manager" class="hash-link"></a></Link></dt>
<dd>
Builds an IO manager definition that reads inputs from and writes outputs to ClickHouse.
`TableSlice.schema` is mapped to a ClickHouse <strong>database</strong> name (not a separate schema layer).
Parameters:
- <strong>type_handlers</strong> – Handlers defining how to translate between ClickHouse tables and in-memory types (e.g. a Pandas `DataFrame`). If only one handler is provided, it becomes the default load type when no annotation is present.
- <strong>default_load_type</strong> – Type to use when an input has no type annotation.
Returns: IOManagerDefinition
</dd>
</dl>
</div>
<div class="section" id="database-utilities">
## Database utilities
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.ClickhouseDbClient'>`class` dagster_clickhouse.ClickhouseDbClient<a href="#dagster_clickhouse.ClickhouseDbClient" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.format_clickhouse_table_fqn'>dagster_clickhouse.format_clickhouse_table_fqn<a href="#dagster_clickhouse.format_clickhouse_table_fqn" class="hash-link"></a></Link></dt>
<dd>
Return the fully quoted `database`.`table` name (Dagster `schema` = ClickHouse database).
</dd>
</dl>
</div>
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_clickhouse.ClickhouseQueryComponent'>`class` dagster_clickhouse.ClickhouseQueryComponent<a href="#dagster_clickhouse.ClickhouseQueryComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A ClickHouse connection component for `TemplatedSqlComponent` and other SQL tooling.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-dask library'
sidebar_position: 1000
title_meta: 'dagster-dask library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-dask library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-dask-library">
# dagster-dask library
See also the [Dask deployment guide](https://docs.dagster.io/deployment/execution/dask).
<dl>
<dt><Link class="anchor" id='dagster_dask.dask_executor'>dagster_dask.dask_executor ExecutorDefinition<a href="#dagster_dask.dask_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Dask-based executor.
The ‘cluster’ can be one of the following:
(‘existing’, ‘local’, ‘yarn’, ‘ssh’, ‘pbs’, ‘moab’, ‘sge’, ‘lsf’, ‘slurm’, ‘oar’, ‘kube’).
If the Dask executor is used without providing executor-specific config, a local Dask cluster
will be created (as when calling `dask.distributed.Client()`
with `dask.distributed.LocalCluster()`).
The Dask executor optionally takes the following config:
```none
cluster:
{
local?: # takes distributed.LocalCluster parameters
{
timeout?: 5, # Timeout duration for initial connection to the scheduler
n_workers?: 4 # Number of workers to start
threads_per_worker?: 1 # Number of threads per each worker
}
}
```
To use the <cite>dask_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster import job
from dagster_dask import dask_executor
@job(executor_def=dask_executor)
def dask_enabled_job():
pass
```
</dd>
</dl>
</div>
---
---
title: 'dagster-databricks library'
sidebar_position: 1000
title_meta: 'dagster-databricks library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-databricks library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-databricks-library">
# dagster-databricks library
The `dagster_databricks` package provides these main pieces of functionality:
- A resource, `databricks_pyspark_step_launcher`, which will execute a op within a Databricks context on a cluster, such that the `pyspark` resource uses the cluster’s Spark instance.
- An op factory, `create_databricks_run_now_op`, which creates an op that launches an existing Databricks job using the [Run Now API](https://docs.databricks.com/api/workspace/jobs/runnow).
- A op factory, `create_databricks_submit_run_op`, which creates an op that submits a one-time run of a set of tasks on Databricks using the [Submit Run API](https://docs.databricks.com/api/workspace/jobs/submit).
Note that, for the `databricks_pyspark_step_launcher`, either S3 or Azure Data Lake Storage config
<strong>must</strong> be specified for ops to succeed, and the credentials for this storage must also be
stored as a Databricks Secret and stored in the resource config so that the Databricks cluster can
access storage.
<div class="section" id="components">
## Components
<div class="section" id="databricksassetbundlecomponent">
### DatabricksAssetBundleComponent
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksAssetBundleComponent'>`class` dagster_databricks.DatabricksAssetBundleComponent<a href="#dagster_databricks.DatabricksAssetBundleComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksAssetBundleComponent.get_asset_spec'>get_asset_spec<a href="#dagster_databricks.DatabricksAssetBundleComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Databricks task.
This method can be overridden in a subclass to customize how Databricks Asset Bundle
tasks are converted to Dagster asset specs. By default, it creates an asset spec with
metadata about the task type, configuration, and dependencies.
Parameters: <strong>task</strong> – The DatabricksBaseTask containing information about the Databricks job taskReturns: An AssetSpec that represents the Databricks task as a Dagster asset
Example:
Override this method to add custom tags or modify the asset key:
```python
from dagster_databricks import DatabricksAssetBundleComponent
from dagster import AssetSpec
class CustomDatabricksAssetBundleComponent(DatabricksAssetBundleComponent):
def get_asset_spec(self, task):
base_spec = super().get_asset_spec(task)
return base_spec.replace_attributes(
tags={
**base_spec.tags,
"job_name": task.job_name,
"environment": "production"
}
)
```
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="databricksworkspacecomponent">
### DatabricksWorkspaceComponent
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksWorkspaceComponent'>`class` dagster_databricks.DatabricksWorkspaceComponent<a href="#dagster_databricks.DatabricksWorkspaceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Component that fetches Databricks workspace jobs and exposes them as assets.
</dd>
</dl>
</div></div>
<div class="section" id="apis">
## APIs
<div class="section" id="resources">
### Resources
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksClientResource'>dagster_databricks.DatabricksClientResource ResourceDefinition<a href="#dagster_databricks.DatabricksClientResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource which provides a Python client for interacting with Databricks within an
op or asset.
Supports four mutually exclusive authentication methods:
- <strong>PAT</strong>: Provide `token`.
- <strong>OAuth M2M</strong>: Provide `oauth_credentials` (client ID + secret).
- <strong>Azure service principal</strong>: Provide `azure_credentials`.
- <strong>Custom CredentialsStrategy</strong>: Pass a `credentials_strategy` instance to the constructor. This supports any authentication flow backed by the Databricks SDK’s `CredentialsStrategy` protocol, including OIDC federation, external IdP token exchange, and other custom auth flows. Because `CredentialsStrategy` is not a serializable config type, it is accepted as a constructor argument and stored outside of Dagster’s config schema machinery.
Examples:
```python
from dagster import job
from dagster_databricks import DatabricksClientResource
from my_auth import MyCredentialsStrategy
@job(
resource_defs={
"databricks": DatabricksClientResource(
host="https://my-workspace.cloud.databricks.com",
credentials_strategy=MyCredentialsStrategy(),
)
}
)
def my_job():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksClient'>`class` dagster_databricks.DatabricksClient<a href="#dagster_databricks.DatabricksClient" class="hash-link"></a></Link></dt>
<dd>
A thin wrapper over the Databricks REST API.
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksClient.workspace_client'>`property` workspace_client<a href="#dagster_databricks.DatabricksClient.workspace_client" class="hash-link"></a></Link></dt>
<dd>
Retrieve a reference to the underlying Databricks Workspace client. For more information,
see the [Databricks SDK for Python](https://docs.databricks.com/dev-tools/sdk-python.html).
<strong>Examples:</strong>
```python
from dagster import op
from databricks.sdk import WorkspaceClient
@op(required_resource_keys={"databricks_client"})
def op1(context):
# Initialize the Databricks Jobs API
client = context.resources.databricks_client.api_client
# Example 1: Run a Databricks job with some parameters.
client.jobs.run_now(...)
# Example 2: Trigger a one-time run of a Databricks workload.
client.jobs.submit(...)
# Example 3: Get an existing run.
client.jobs.get_run(...)
# Example 4: Cancel a run.
client.jobs.cancel_run(...)
```
Returns: The authenticated Databricks SDK Workspace Client.Return type: WorkspaceClient
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="ops">
### Ops
<dl>
<dt><Link class="anchor" id='dagster_databricks.create_databricks_run_now_op'>dagster_databricks.create_databricks_run_now_op<a href="#dagster_databricks.create_databricks_run_now_op" class="hash-link"></a></Link></dt>
<dd>
Creates an op that launches an existing databricks job.
As config, the op accepts a blob of the form described in Databricks’ Job API:
[https://docs.databricks.com/api/workspace/jobs/runnow](https://docs.databricks.com/api/workspace/jobs/runnow). The only required field is
`job_id`, which is the ID of the job to be executed. Additional fields can be used to specify
override parameters for the Databricks Job.
Parameters:
- <strong>databricks_job_id</strong> (<em>int</em>) – The ID of the Databricks Job to be executed.
- <strong>databricks_job_configuration</strong> (<em>dict</em>) – Configuration for triggering a new job run of a Databricks Job. See [https://docs.databricks.com/api/workspace/jobs/runnow](https://docs.databricks.com/api/workspace/jobs/runnow) for the full configuration.
- <strong>poll_interval_seconds</strong> (<em>float</em>) – How often to poll the Databricks API to check whether the Databricks job has finished running.
- <strong>max_wait_time_seconds</strong> (<em>float</em>) – How long to wait for the Databricks job to finish running before raising an error.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op. If not provided, the name will be _databricks_run_now_op.
- <strong>databricks_resource_key</strong> (<em>str</em>) – The name of the resource key used by this op. If not provided, the resource key will be “databricks”.
Returns: An op definition to run the Databricks Job.Return type: [OpDefinition](../../../api/dagster/ops.mdx#dagster.OpDefinition)
Example:
```python
from dagster import job
from dagster_databricks import create_databricks_run_now_op, DatabricksClientResource
DATABRICKS_JOB_ID = 1234
run_now_op = create_databricks_run_now_op(
databricks_job_id=DATABRICKS_JOB_ID,
databricks_job_configuration={
"python_params": [
"--input",
"schema.db.input_table",
"--output",
"schema.db.output_table",
],
},
)
@job(
resource_defs={
"databricks": DatabricksClientResource(
host=EnvVar("DATABRICKS_HOST"),
token=EnvVar("DATABRICKS_TOKEN")
)
}
)
def do_stuff():
run_now_op()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_databricks.create_databricks_submit_run_op'>dagster_databricks.create_databricks_submit_run_op<a href="#dagster_databricks.create_databricks_submit_run_op" class="hash-link"></a></Link></dt>
<dd>
Creates an op that submits a one-time run of a set of tasks on Databricks.
As config, the op accepts a blob of the form described in Databricks’ Job API:
[https://docs.databricks.com/api/workspace/jobs/submit](https://docs.databricks.com/api/workspace/jobs/submit).
Parameters:
- <strong>databricks_job_configuration</strong> (<em>dict</em>) – Configuration for submitting a one-time run of a set of tasks on Databricks. See [https://docs.databricks.com/api/workspace/jobs/submit](https://docs.databricks.com/api/workspace/jobs/submit) for the full configuration.
- <strong>poll_interval_seconds</strong> (<em>float</em>) – How often to poll the Databricks API to check whether the Databricks job has finished running.
- <strong>max_wait_time_seconds</strong> (<em>float</em>) – How long to wait for the Databricks job to finish running before raising an error.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op. If not provided, the name will be _databricks_submit_run_op.
- <strong>databricks_resource_key</strong> (<em>str</em>) – The name of the resource key used by this op. If not provided, the resource key will be “databricks”.
Returns: An op definition to submit a one-time run of a set of tasks on Databricks.Return type: [OpDefinition](../../../api/dagster/ops.mdx#dagster.OpDefinition)
Example:
```python
from dagster import job
from dagster_databricks import create_databricks_submit_run_op, DatabricksClientResource
submit_run_op = create_databricks_submit_run_op(
databricks_job_configuration={
"new_cluster": {
"spark_version": '2.1.0-db3-scala2.11',
"num_workers": 2
},
"notebook_task": {
"notebook_path": "/Users/dagster@example.com/PrepareData",
},
}
)
@job(
resource_defs={
"databricks": DatabricksClientResource(
host=EnvVar("DATABRICKS_HOST"),
token=EnvVar("DATABRICKS_TOKEN")
)
}
)
def do_stuff():
submit_run_op()
```
</dd>
</dl>
</div>
<div class="section" id="step-launcher">
### Step Launcher
<dl>
<dt><Link class="anchor" id='dagster_databricks.databricks_pyspark_step_launcher'>dagster_databricks.databricks_pyspark_step_launcher ResourceDefinition<a href="#dagster_databricks.databricks_pyspark_step_launcher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning superseded
This API has been superseded.
While there is no plan to remove this functionality, for new projects, we recommend using Dagster Pipes. For more information, see https://docs.dagster.io/integrations/external-pipelines.
:::
Resource for running ops as a Databricks Job.
When this resource is used, the op will be executed in Databricks using the ‘Run Submit’
API. Pipeline code will be zipped up and copied to a directory in DBFS along with the op’s
execution context.
Use the ‘run_config’ configuration to specify the details of the Databricks cluster used, and
the ‘storage’ key to configure persistent storage on that cluster. Storage is accessed by
setting the credentials in the Spark context, as documented [here for S3](https://docs.databricks.com/data/data-sources/aws/amazon-s3.html#alternative-1-set-aws-keys-in-the-spark-context) and [here for ADLS](https://docs.microsoft.com/en-gb/azure/databricks/data/data-sources/azure/azure-datalake-gen2#--access-directly-using-the-storage-account-access-key).
</dd>
</dl>
</div>
<div class="section" id="pipes">
### Pipes
<dl>
<dt><Link class="anchor" id='dagster_databricks.PipesDatabricksClient'>`class` dagster_databricks.PipesDatabricksClient<a href="#dagster_databricks.PipesDatabricksClient" class="hash-link"></a></Link></dt>
<dd>
Pipes client for databricks.
Parameters:
- <strong>client</strong> (<em>WorkspaceClient</em>) – A databricks <cite>WorkspaceClient</cite> object.
- <strong>(</strong><strong>Optional</strong><strong>[</strong><strong>Mapping</strong><strong>[</strong><strong>str</strong> (<em>env</em>) – An optional dict of environment variables to pass to the databricks job.
- <strong>str</strong><strong>]</strong><strong>]</strong> – An optional dict of environment variables to pass to the databricks job.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the k8s container process. Defaults to [`PipesDbfsContextInjector`](#dagster_databricks.PipesDbfsContextInjector).
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the databricks job. Defaults to [`PipesDbfsMessageReader`](#dagster_databricks.PipesDbfsMessageReader).
- <strong>poll_interval_seconds</strong> (<em>float</em>) – How long to sleep between checking the status of the job run. Defaults to 5.
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the Databricks job if the orchestration process is interrupted or canceled. Defaults to True.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_databricks.PipesDbfsContextInjector'>`class` dagster_databricks.PipesDbfsContextInjector<a href="#dagster_databricks.PipesDbfsContextInjector" class="hash-link"></a></Link></dt>
<dd>
A context injector that injects context into a Databricks job by writing a JSON file to DBFS.
Parameters: <strong>client</strong> (<em>WorkspaceClient</em>) – A databricks <cite>WorkspaceClient</cite> object.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_databricks.PipesDbfsMessageReader'>`class` dagster_databricks.PipesDbfsMessageReader<a href="#dagster_databricks.PipesDbfsMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by periodically reading message chunks from an
automatically-generated temporary directory on DBFS.
If <cite>log_readers</cite> is passed, this reader will also start the passed readers
when the first message is received from the external process.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a chunk
- <strong>client</strong> (<em>WorkspaceClient</em>) – A databricks <cite>WorkspaceClient</cite> object.
- <strong>cluster_log_root</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The root path on DBFS where the cluster logs are written. If set, this will be used to read stderr/stdout logs.
- <strong>include_stdio_in_messages</strong> (<em>bool</em>) – Whether to send stdout/stderr to Dagster via Pipes messages. Defaults to False.
- <strong>log_readers</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>PipesLogReader</em><em>]</em><em>]</em>) – A set of log readers for logs on DBFS.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_databricks.PipesDbfsLogReader'>`class` dagster_databricks.PipesDbfsLogReader<a href="#dagster_databricks.PipesDbfsLogReader" class="hash-link"></a></Link></dt>
<dd>
Reader that reads a log file from DBFS.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a log chunk
- <strong>remote_log_name</strong> (<em>Literal</em><em>[</em><em>"stdout"</em><em>, </em><em>"stderr"</em><em>]</em>) – The name of the log file to read.
- <strong>target_stream</strong> (<em>TextIO</em>) – The stream to which to forward log chunks that have been read.
- <strong>client</strong> (<em>WorkspaceClient</em>) – A databricks <cite>WorkspaceClient</cite> object.
- <strong>debug_info</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional message containing debug information about the log reader.
</dd>
</dl>
</div>
<div class="section" id="error">
### Error
<dl>
<dt><Link class="anchor" id='dagster_databricks.DatabricksError'>`class` dagster_databricks.DatabricksError<a href="#dagster_databricks.DatabricksError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
<div class="section" id="legacy">
### Legacy
<dl>
<dt><Link class="anchor" id='dagster_databricks.databricks_client'>dagster_databricks.databricks_client ResourceDefinition<a href="#dagster_databricks.databricks_client" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-datadog library'
sidebar_position: 1000
title_meta: 'dagster-datadog library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-datadog library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-datadog-library">
# dagster-datadog library
This library provides an integration with Datadog, to support publishing metrics to Datadog from
within Dagster ops.
We use the Python [datadogpy](https://github.com/DataDog/datadogpy) library. To use it, you’ll
first need to create a DataDog account and get both [API and Application keys](https://docs.datadoghq.com/account_management/api-app-keys).
The integration uses [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd), so you’ll need
to ensure the datadog agent is running on the host you’re sending metrics from.
<dl>
<dt><Link class="anchor" id='dagster_datadog.DatadogResource'>dagster_datadog.DatadogResource ResourceDefinition<a href="#dagster_datadog.DatadogResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource is a thin wrapper over the
[dogstatsd library](https://datadogpy.readthedocs.io/en/latest/).
As such, we directly mirror the public API methods of DogStatsd here;
for more information on how to use this resource, see the
[Datadog documentation](https://docs.datadoghq.com/developers/dogstatsd/).
Examples:
```python
@op
def datadog_op(datadog_resource: DatadogResource):
datadog_client = datadog_resource.get_client()
datadog_client.event('Man down!', 'This server needs assistance.')
datadog_client.gauge('users.online', 1001, tags=["protocol:http"])
datadog_client.increment('page.views')
datadog_client.decrement('page.views')
datadog_client.histogram('album.photo.count', 26, tags=["gender:female"])
datadog_client.distribution('album.photo.count', 26, tags=["color:blue"])
datadog_client.set('visitors.uniques', 999, tags=["browser:ie"])
datadog_client.service_check('svc.check_name', datadog_client.WARNING)
datadog_client.timing("query.response.time", 1234)
# Use timed decorator
@datadog_client.timed('run_fn')
def run_fn():
pass
run_fn()
@job
def job_for_datadog_op() -> None:
datadog_op()
job_for_datadog_op.execute_in_process(
resources={"datadog_resource": DatadogResource(api_key="FOO", app_key="BAR")}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_datadog.datadog_resource'>dagster_datadog.datadog_resource ResourceDefinition<a href="#dagster_datadog.datadog_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This legacy resource is a thin wrapper over the
[dogstatsd library](https://datadogpy.readthedocs.io/en/latest/).
Prefer using [`DatadogResource`](#dagster_datadog.DatadogResource).
As such, we directly mirror the public API methods of DogStatsd here; you can refer to the
[DataDog documentation](https://docs.datadoghq.com/developers/dogstatsd/) for how to use this
resource.
Examples:
```python
@op(required_resource_keys={'datadog'})
def datadog_op(context):
dd = context.resources.datadog
dd.event('Man down!', 'This server needs assistance.')
dd.gauge('users.online', 1001, tags=["protocol:http"])
dd.increment('page.views')
dd.decrement('page.views')
dd.histogram('album.photo.count', 26, tags=["gender:female"])
dd.distribution('album.photo.count', 26, tags=["color:blue"])
dd.set('visitors.uniques', 999, tags=["browser:ie"])
dd.service_check('svc.check_name', dd.WARNING)
dd.timing("query.response.time", 1234)
# Use timed decorator
@dd.timed('run_fn')
def run_fn():
pass
run_fn()
@job(resource_defs={'datadog': datadog_resource})
def dd_job():
datadog_op()
result = dd_job.execute_in_process(
run_config={'resources': {'datadog': {'config': {'api_key': 'YOUR_KEY', 'app_key': 'YOUR_KEY'}}}}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-datahub library'
sidebar_position: 1000
title_meta: 'dagster-datahub library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-datahub library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-datahub-library">
# dagster-datahub library
This library provides an integration with [Datahub](https://datahub.com/) to support pushing metadata to Datahub from
within Dagster ops.
<div className='lineblock'> </div>
We use the [Datahub Python Library](https://github.com/datahub-project/datahub). To use it, you’ll
first need to start up a Datahub Instance. For more information, see the [Datahub Quickstart Guide](https://datahubproject.io/docs/quickstart).
<div className='lineblock'> </div>
<dl>
<dt><Link class="anchor" id='dagster_datahub.DatahubRESTEmitterResource'>dagster_datahub.DatahubRESTEmitterResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-datahub/dagster_datahub/resources.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_datahub.DatahubRESTEmitterResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for Dagster resources that utilize structured config.
This class is a subclass of both `ResourceDefinition` and `Config`.
Example definition:
```python
class WriterResource(ConfigurableResource):
prefix: str
def output(self, text: str) -> None:
print(f"{self.prefix}{text}")
```
Example usage:
```python
@asset
def asset_that_uses_writer(writer: WriterResource):
writer.output("text")
defs = Definitions(
assets=[asset_that_uses_writer],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
You can optionally use this class to model configuration only and vend an object
of a different type for use at runtime. This is useful for those who wish to
have a separate object that manages configuration and a separate object at runtime. Or
where you want to directly use a third-party class that you do not control.
To do this you override the <cite>create_resource</cite> methods to return a different object.
```python
class WriterResource(ConfigurableResource):
prefix: str
def create_resource(self, context: InitResourceContext) -> Writer:
# Writer is pre-existing class defined else
return Writer(self.prefix)
```
Example usage:
```python
@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
writer.output("text")
defs = Definitions(
assets=[use_preexisting_writer_as_resource],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_datahub.DatahubKafkaEmitterResource'>dagster_datahub.DatahubKafkaEmitterResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-datahub/dagster_datahub/resources.py#L78' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_datahub.DatahubKafkaEmitterResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for Dagster resources that utilize structured config.
This class is a subclass of both `ResourceDefinition` and `Config`.
Example definition:
```python
class WriterResource(ConfigurableResource):
prefix: str
def output(self, text: str) -> None:
print(f"{self.prefix}{text}")
```
Example usage:
```python
@asset
def asset_that_uses_writer(writer: WriterResource):
writer.output("text")
defs = Definitions(
assets=[asset_that_uses_writer],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
You can optionally use this class to model configuration only and vend an object
of a different type for use at runtime. This is useful for those who wish to
have a separate object that manages configuration and a separate object at runtime. Or
where you want to directly use a third-party class that you do not control.
To do this you override the <cite>create_resource</cite> methods to return a different object.
```python
class WriterResource(ConfigurableResource):
prefix: str
def create_resource(self, context: InitResourceContext) -> Writer:
# Writer is pre-existing class defined else
return Writer(self.prefix)
```
Example usage:
```python
@asset
def use_preexisting_writer_as_resource(writer: ResourceParam[Writer]):
writer.output("text")
defs = Definitions(
assets=[use_preexisting_writer_as_resource],
resources={"writer": WriterResource(prefix="a_prefix")},
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_datahub.datahub_rest_emitter'>dagster_datahub.datahub_rest_emitter ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-datahub/dagster_datahub/resources.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_datahub.datahub_rest_emitter" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_datahub.datahub_kafka_emitter'>dagster_datahub.datahub_kafka_emitter ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-datahub/dagster_datahub/resources.py#L99' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_datahub.datahub_kafka_emitter" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
</div></div>
---
---
title: 'dagster-dbt library'
sidebar_position: 1000
title_meta: 'dagster-dbt library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-dbt library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-dbt-library">
# dagster-dbt library
Dagster orchestrates [dbt](https://www.getdbt.com/) alongside other technologies, so you can combine dbt with Spark, Python,
and other tools in a single workflow. Dagster’s software-defined asset abstractions make it simple to define
data assets that depend on specific dbt models, or define the computation required to compute
the sources that your dbt models depend on.
For more information on using the dbt and dbt Cloud integrations, see the [Dagster & dbt](https://docs.dagster.io/integrations/libraries/dbt) and
[Dagster & dbt Cloud](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud) docs.
<div class="section" id="dbtprojectcomponent">
## DbtProjectComponent
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtProjectComponent'>`class` dagster_dbt.DbtProjectComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtProjectComponent" class="hash-link"></a></Link></dt>
<dd>
Expose a DBT project to Dagster as a set of assets.
This component assumes that you have already set up a dbt project, for example, the dbt [Jaffle shop](https://github.com/dbt-labs/jaffle-shop). Run <cite>git clone –depth=1 https://github.com/dbt-labs/jaffle-shop.git jaffle_shop && rm -rf jaffle_shop/.git</cite> to copy that project
into your Dagster project directory.
Scaffold a DbtProjectComponent definition by running <cite>dg scaffold defs dagster_dbt.DbtProjectComponent –project-path path/to/your/existing/dbt_project</cite>
in the Dagster project directory.
Example:
```yaml
# defs.yaml
type: dagster_dbt.DbtProjectComponent
attributes:
project: "{{ project_root }}/path/to/dbt_project"
cli_args:
- build
```
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtProjectComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py#L398' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtProjectComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes the dbt command for the selected assets.
This method can be overridden in a subclass to customize the execution behavior,
such as adding custom logging, modifying CLI arguments, or handling events differently.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>dbt</strong> – The DbtCliResource used to execute dbt commands
Yields: Events from the dbt CLI execution (e.g., AssetMaterialization, AssetObservation)
Example:
Override this method to add custom logging before and after execution:
```python
from dagster_dbt import DbtProjectComponent
import dagster as dg
class CustomDbtProjectComponent(DbtProjectComponent):
def execute(self, context, dbt):
context.log.info("Starting custom dbt execution")
yield from super().execute(context, dbt)
context.log.info("Completed custom dbt execution")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtProjectComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py#L281' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtProjectComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given dbt node.
This method can be overridden in a subclass to customize how dbt nodes are converted
to Dagster asset specs. By default, it delegates to the configured DagsterDbtTranslator.
Parameters:
- <strong>manifest</strong> – The dbt manifest dictionary containing information about all dbt nodes
- <strong>unique_id</strong> – The unique identifier for the dbt node (e.g., “model.my_project.my_model”)
- <strong>project</strong> – The DbtProject object, if available
Returns: An AssetSpec that represents the dbt node as a Dagster asset
Example:
Override this method to add custom tags to all dbt models:
```python
from dagster_dbt import DbtProjectComponent
import dagster as dg
class CustomDbtProjectComponent(DbtProjectComponent):
def get_asset_spec(self, manifest, unique_id, project):
base_spec = super().get_asset_spec(manifest, unique_id, project)
return base_spec.replace_attributes(
tags={**base_spec.tags, "custom_tag": "my_value"}
)
```
</dd>
</dl>
</dd>
</dl>
To use the dbt component, see the [dbt component integration guide](https://docs.dagster.io/integrations/libraries/dbt).
<div class="section" id="component-yaml">
### Component YAML
When you scaffold a dbt component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_dbt.DbtProjectComponent
attributes:
project: '{{ context.project_root }}/dbt'
```
</div></div>
<div class="section" id="dbtcloudcomponent">
## DbtCloudComponent
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudComponent'>`class` dagster_dbt.DbtCloudComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/component/dbt_cloud_component.py#L124' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudComponent" class="hash-link"></a></Link></dt>
<dd>
Expose a dbt Cloud workspace to Dagster as a set of assets.
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/component/dbt_cloud_component.py#L245' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given dbt node.
This method can be overridden in a subclass to customize how dbt nodes are converted
to Dagster asset specs. By default, it delegates to the configured DagsterDbtTranslator.
Parameters:
- <strong>manifest</strong> – The dbt manifest dictionary containing information about all dbt nodes.
- <strong>unique_id</strong> – The unique identifier for the dbt node (e.g., “model.my_project.my_model”).
- <strong>project</strong> – Always `None` for dbt Cloud (execution is remote).
Returns: An AssetSpec that represents the dbt node as a Dagster asset.
Example:
```python
from dagster_dbt import DbtCloudComponent
import dagster as dg
class MyDbtCloudComponent(DbtCloudComponent):
def get_asset_spec(self, manifest, unique_id, project):
base_spec = super().get_asset_spec(manifest, unique_id, project)
return base_spec.replace_attributes(
tags={**base_spec.tags, "custom_tag": "my_value"}
)
```
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="id1">
## dagster-dbt
<div class="section" id="dagster-dbt-project">
### dagster-dbt project
Commands for using a dbt project in Dagster.
```shell
dagster-dbt project [OPTIONS] COMMAND [ARGS]...
```
<div class="section" id="dagster-dbt-project-prepare-and-package">
#### prepare-and-package
This command will invoke `prepare_and_package` on [`DbtProject`](#dagster_dbt.DbtProject) found in the target module or file.
Note that this command runs <cite>dbt deps</cite> and <cite>dbt parse</cite>.
```shell
dagster-dbt project prepare-and-package [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-dbt-project-prepare-and-package-file'>--file \<file><a href="#cmdoption-dagster-dbt-project-prepare-and-package-file" class="hash-link"></a></Link></dt>
<dd>
The file containing DbtProject definitions to prepare.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-dbt-project-prepare-and-package-components'>--components \<components><a href="#cmdoption-dagster-dbt-project-prepare-and-package-components" class="hash-link"></a></Link></dt>
<dd>
The path to a dg project directory containing DbtProjectComponents.
</dd>
</dl>
</div>
<div class="section" id="dagster-dbt-project-scaffold">
#### scaffold
This command will initialize a new Dagster project and create directories and files that
load assets from an existing dbt project.
```shell
dagster-dbt project scaffold [OPTIONS]
```
Options:
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-dbt-project-scaffold-project-name'>--project-name \<project_name><a href="#cmdoption-dagster-dbt-project-scaffold-project-name" class="hash-link"></a></Link></dt>
<dd>
<strong>Required</strong> The name of the Dagster project to initialize for your dbt project.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='cmdoption-dagster-dbt-project-scaffold-dbt-project-dir'>--dbt-project-dir \<dbt_project_dir><a href="#cmdoption-dagster-dbt-project-scaffold-dbt-project-dir" class="hash-link"></a></Link></dt>
<dd>
The path of your dbt project directory. This path must contain a dbt_project.yml file. By default, this command will assume that the current working directory contains a dbt project, but you can set a different directory by setting this option.
</dd>
</dl>
</div></div></div>
<div class="section" id="dbt-core">
## dbt Core
Here, we provide interfaces to manage dbt projects invoked by the local dbt command line interface
(dbt CLI).
<div class="section" id="assets-dbt-core">
### Assets (dbt Core)
<dl>
<dt><Link class="anchor" id='dagster_dbt.dbt_assets'>@dagster_dbt.dbt_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_decorator.py#L29' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.dbt_assets" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to compute a set of dbt resources, described by a manifest.json.
When invoking dbt commands using [`DbtCliResource`](#dagster_dbt.DbtCliResource)’s
[`cli()`](#dagster_dbt.DbtCliResource.cli) method, Dagster events are emitted by calling
`yield from` on the event stream returned by [`stream()`](#dagster_dbt.DbtCliInvocation.stream).
Parameters:
- <strong>manifest</strong> (<em>Union</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>, </em><em>str</em><em>, </em><em>Path</em><em>]</em>) – The contents of a manifest.json file or the path to a manifest.json file. A manifest.json contains a representation of a dbt project (models, tests, macros, etc). We use this representation to create corresponding Dagster assets.
- <strong>select</strong> (<em>str</em>) – A dbt selection string for the models in a project that you want to include. Defaults to `fqn:*`.
- <strong>exclude</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A dbt selection string for the models in a project that you want to exclude. Defaults to “”.
- <strong>selector</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A dbt selector for the models in a project that you want to include. Cannot be combined with select or exclude. Defaults to None.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the op.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The IO manager key that will be set on each of the returned assets. When other ops are downstream of the loaded assets, the IOManager specified here determines how the inputs to those ops are loaded. Defaults to “io_manager”.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the dbt assets.
- <strong>dagster_dbt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDbtTranslator*](#dagster_dbt.DagsterDbtTranslator)<em>]</em>) – Allows customizing how to map dbt models, seeds, etc. to asset keys and asset metadata.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](../../../api/dagster/partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – If a partitions_def is defined, this determines how to execute backfills that target multiple partitions. If a time window partition definition is used, this parameter defaults to a single-run policy.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the assets. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of required resource handles.
- <strong>project</strong> (<em>Optional</em><em>[</em>[*DbtProject*](#dagster_dbt.DbtProject)<em>]</em>) – A DbtProject instance which provides a pointer to the dbt project location and manifest. Not required, but needed to attach code references from model code to Dagster assets.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](../../../api/dagster/ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that computes the asset.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs the dbt assets’ execution.
Examples:
Running `dbt build` for a dbt project:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
```
Running dbt commands with flags:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build", "--full-refresh"], context=context).stream()
```
Running dbt commands with `--vars`:
```python
import json
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_vars = {"key": "value"}
yield from dbt.cli(["build", "--vars", json.dumps(dbt_vars)], context=context).stream()
```
Retrieving dbt artifacts after running a dbt command:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_build_invocation = dbt.cli(["build"], context=context)
yield from dbt_build_invocation.stream()
run_results_json = dbt_build_invocation.get_artifact("run_results.json")
```
Running multiple dbt commands for a dbt project:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["run"], context=context).stream()
yield from dbt.cli(["test"], context=context).stream()
```
Accessing the dbt event stream alongside the Dagster event stream:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_cli_invocation = dbt.cli(["build"], context=context)
# Each dbt event is structured: https://docs.getdbt.com/reference/events-logging
for dbt_event in dbt_invocation.stream_raw_events():
for dagster_event in dbt_event.to_default_asset_events(
manifest=dbt_invocation.manifest,
dagster_dbt_translator=dbt_invocation.dagster_dbt_translator,
context=dbt_invocation.context,
target_path=dbt_invocation.target_path,
):
# Manipulate `dbt_event`
...
# Then yield the Dagster event
yield dagster_event
```
Customizing the Dagster asset definition metadata inferred from a dbt project using [`DagsterDbtTranslator`](#dagster_dbt.DagsterDbtTranslator):
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, dbt_assets
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
...
@dbt_assets(
manifest=Path("target", "manifest.json"),
dagster_dbt_translator=CustomDagsterDbtTranslator(),
)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
```
Using a custom resource key for dbt:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, my_custom_dbt_resource_key: DbtCliResource):
yield from my_custom_dbt_resource_key.cli(["build"], context=context).stream()
```
Using a dynamically generated resource key for dbt using <cite>required_resource_keys</cite>:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
dbt_resource_key = "my_custom_dbt_resource_key"
@dbt_assets(manifest=Path("target", "manifest.json"), required_resource_keys={my_custom_dbt_resource_key})
def my_dbt_assets(context: AssetExecutionContext):
dbt = getattr(context.resources, dbt_resource_key)
yield from dbt.cli(["build"], context=context).stream()
```
Invoking another Dagster [`ResourceDefinition`](../../../api/dagster/resources.mdx#dagster.ResourceDefinition) alongside dbt:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
from dagster_slack import SlackResource
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource, slack: SlackResource):
yield from dbt.cli(["build"], context=context).stream()
slack_client = slack.get_client()
slack_client.chat_postMessage(channel="#my-channel", text="dbt build succeeded!")
```
Defining and accessing Dagster [`Config`](../../../api/dagster/config.mdx#dagster.Config) alongside dbt:
```python
from pathlib import Path
from dagster import AssetExecutionContext, Config
from dagster_dbt import DbtCliResource, dbt_assets
class MyDbtConfig(Config):
full_refresh: bool
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource, config: MyDbtConfig):
dbt_build_args = ["build"]
if config.full_refresh:
dbt_build_args += ["--full-refresh"]
yield from dbt.cli(dbt_build_args, context=context).stream()
```
Defining Dagster `PartitionDefinition` alongside dbt:
```python
import json
from pathlib import Path
from dagster import AssetExecutionContext, DailyPartitionDefinition
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(
manifest=Path("target", "manifest.json"),
partitions_def=DailyPartitionsDefinition(start_date="2023-01-01")
)
def partitionshop_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
time_window = context.partition_time_window
dbt_vars = {
"min_date": time_window.start.isoformat(),
"max_date": time_window.end.isoformat()
}
dbt_build_args = ["build", "--vars", json.dumps(dbt_vars)]
yield from dbt.cli(dbt_build_args, context=context).stream()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator'>`class` dagster_dbt.DagsterDbtTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L84' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator" class="hash-link"></a></Link></dt>
<dd>
Holds a set of methods that derive Dagster asset definition metadata given a representation
of a dbt resource (models, tests, sources, etc).
This class is exposed so that methods can be overriden to customize how Dagster asset metadata
is derived.
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_asset_key'>get_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L324' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_asset_key" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster asset key that represents that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom asset key for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: The Dagster asset key for the dbt resource.Return type: [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)
Examples:
Adding a prefix to the default asset key generated for each dbt resource:
```python
from typing import Any, Mapping
from dagster import AssetKey
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_asset_key(self, dbt_resource_props: Mapping[str, Any]) -> AssetKey:
return super().get_asset_key(dbt_resource_props).with_prefix("prefix")
```
Adding a prefix to the default asset key generated for each dbt resource, but only for dbt sources:
```python
from typing import Any, Mapping
from dagster import AssetKey
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_asset_key(self, dbt_resource_props: Mapping[str, Any]) -> AssetKey:
asset_key = super().get_asset_key(dbt_resource_props)
if dbt_resource_props["resource_type"] == "source":
asset_key = asset_key.with_prefix("my_prefix")
return asset_key
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_auto_materialize_policy'>get_auto_materialize_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L609' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_auto_materialize_policy" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster `dagster.AutoMaterializePolicy` for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom auto-materialize policy for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A Dagster auto-materialize policy.Return type: Optional[AutoMaterializePolicy]
Examples:
Set a custom auto-materialize policy for all dbt resources:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_auto_materialize_policy(self, dbt_resource_props: Mapping[str, Any]) -> Optional[AutoMaterializePolicy]:
return AutoMaterializePolicy.eager()
```
Set a custom auto-materialize policy for dbt resources with a specific tag:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_auto_materialize_policy(self, dbt_resource_props: Mapping[str, Any]) -> Optional[AutoMaterializePolicy]:
auto_materialize_policy = None
if "my_custom_tag" in dbt_resource_props.get("tags", []):
auto_materialize_policy = AutoMaterializePolicy.eager()
return auto_materialize_policy
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_automation_condition'>get_automation_condition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L664' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_automation_condition" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster `dagster.AutoMaterializePolicy` for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom AutomationCondition for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A Dagster auto-materialize policy.Return type: Optional[AutoMaterializePolicy]
Examples:
Set a custom AutomationCondition for all dbt resources:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_automation_condition(self, dbt_resource_props: Mapping[str, Any]) -> Optional[AutomationCondition]:
return AutomationCondition.eager()
```
Set a custom AutomationCondition for dbt resources with a specific tag:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_automation_condition(self, dbt_resource_props: Mapping[str, Any]) -> Optional[AutomationCondition]:
automation_condition = None
if "my_custom_tag" in dbt_resource_props.get("tags", []):
automation_condition = AutomationCondition.eager()
return automation_condition
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_code_version'>get_code_version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L545' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_code_version" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster code version for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom code version for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A Dagster code version.Return type: Optional[str]
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_code_version(self, dbt_resource_props: Mapping[str, Any]) -> Optional[str]:
return dbt_resource_props["checksum"]["checksum"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_description'>get_description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L409' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_description" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster description for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom description for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: The description for the dbt resource.Return type: str
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_description(self, dbt_resource_props: Mapping[str, Any]) -> str:
return "custom description"
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_group_name'>get_group_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L513' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_group_name" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster group name for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom group name for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A Dagster group name.Return type: Optional[str]
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_group_name(self, dbt_resource_props: Mapping[str, Any]) -> Optional[str]:
return "custom_group_prefix" + dbt_resource_props.get("config", {}).get("group")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_metadata'>get_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L441' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_metadata" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster metadata for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom metadata for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A dictionary representing the Dagster metadata for the dbt resource.Return type: Mapping[str, Any]
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_metadata(self, dbt_resource_props: Mapping[str, Any]) -> Mapping[str, Any]:
return {"custom": "metadata"}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_owners'>get_owners <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L577' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_owners" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster owners for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide custom owners for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A set of Dagster owners.Return type: Optional[Sequence[str]]
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_owners(self, dbt_resource_props: Mapping[str, Any]) -> Optional[Sequence[str]]:
return ["user@owner.com", "team:team@owner.com"]
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_partition_mapping'>get_partition_mapping <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L378' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_partition_mapping" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A function that takes two dictionaries: the first, representing properties of a dbt
resource; and the second, representing the properties of a parent dependency to the first
dbt resource. The function returns the Dagster partition mapping for the dbt dependency.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
This method can be overridden to provide a custom partition mapping for a dbt dependency.
Parameters:
- <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt child resource.
- <strong>dbt_parent_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt parent resource, in relationship to the child.
Returns: The Dagster partition mapping for the dbt resource. If None is returned, the
default partition mapping will be used.Return type: Optional[[PartitionMapping](../../../api/dagster/partitions.mdx#dagster.PartitionMapping)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslator.get_tags'>get_tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L473' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslator.get_tags" class="hash-link"></a></Link></dt>
<dd>
A function that takes a dictionary representing properties of a dbt resource, and
returns the Dagster tags for that resource.
Note that a dbt resource is unrelated to Dagster’s resource concept, and simply represents
a model, seed, snapshot or source in a given dbt project. You can learn more about dbt
resources and the properties available in this dictionary here:
[https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details](https://docs.getdbt.com/reference/artifacts/manifest-json#resource-details)
dbt tags are strings, but Dagster tags are key-value pairs. To bridge this divide, the dbt
tag string is used as the Dagster tag key, and the Dagster tag value is set to the empty
string, “”.
Any dbt tags that don’t match Dagster’s supported tag key format (e.g. they contain
unsupported characters) will be ignored.
This method can be overridden to provide custom tags for a dbt resource.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.Returns: A dictionary representing the Dagster tags for the dbt resource.Return type: Mapping[str, str]
Examples:
```python
from typing import Any, Mapping
from dagster_dbt import DagsterDbtTranslator
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
def get_tags(self, dbt_resource_props: Mapping[str, Any]) -> Mapping[str, str]:
return {"custom": "tag"}
```
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtTranslatorSettings'>`class` dagster_dbt.DagsterDbtTranslatorSettings <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dagster_dbt_translator.py#L53' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtTranslatorSettings" class="hash-link"></a></Link></dt>
<dd>
Settings to enable Dagster features for your dbt project.
Parameters:
- <strong>enable_asset_checks</strong> (<em>bool</em>) – Whether to load dbt tests as Dagster asset checks. Defaults to True.
- <strong>enable_duplicate_source_asset_keys</strong> (<em>bool</em>) – Whether to allow dbt sources with duplicate Dagster asset keys. Defaults to False.
- <strong>enable_code_references</strong> (<em>bool</em>) – Whether to enable Dagster code references for dbt resources. Defaults to False.
- <strong>enable_dbt_selection_by_name</strong> (<em>bool</em>) – Whether to enable selecting dbt resources by name, rather than fully qualified name. Defaults to False.
- <strong>enable_source_tests_as_checks</strong> (<em>bool</em>) – Whether to load dbt source tests as Dagster asset checks. Defaults to False. If False, asset observations will be emitted for source tests.
- <strong>enable_source_metadata</strong> (<em>bool</em>) – Whether to include metadata on AssetDep objects for dbt sources. If set to True, enables the ability to remap upstream asset keys based on table name. Defaults to True.
- <strong>enable_dbt_views_as_virtual_assets</strong> (<em>bool</em>) – Whether to treat dbt models with `materialized: view` as virtual assets. When enabled, view models will have `is_virtual=True` and `"view"` added to their kinds. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtManifestAssetSelection'>`class` dagster_dbt.DbtManifestAssetSelection <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dbt_manifest_asset_selection.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtManifestAssetSelection" class="hash-link"></a></Link></dt>
<dd>
Defines a selection of assets from a dbt manifest wrapper and a dbt selection string.
Parameters:
- <strong>manifest</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The dbt manifest blob.
- <strong>select</strong> (<em>str</em>) – A dbt selection string to specify a set of dbt resources.
- <strong>exclude</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A dbt selection string to exclude a set of dbt resources.
Examples:
```python
import json
from pathlib import Path
from dagster_dbt import DbtManifestAssetSelection
manifest = json.loads(Path("path/to/manifest.json").read_text())
# select the dbt assets that have the tag "foo".
my_selection = DbtManifestAssetSelection(manifest=manifest, select="tag:foo")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.build_dbt_asset_selection'>dagster_dbt.build_dbt_asset_selection <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L290' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.build_dbt_asset_selection" class="hash-link"></a></Link></dt>
<dd>
Build an asset selection for a dbt selection string.
See [https://docs.getdbt.com/reference/node-selection/syntax#how-does-selection-work](https://docs.getdbt.com/reference/node-selection/syntax#how-does-selection-work) for
more information.
Parameters:
- <strong>dbt_select</strong> (<em>str</em>) – A dbt selection string to specify a set of dbt resources.
- <strong>dbt_exclude</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A dbt selection string to exclude a set of dbt resources.
Returns: An asset selection for the selected dbt nodes.Return type: [AssetSelection](../../../api/dagster/assets.mdx#dagster.AssetSelection)
Examples:
```python
from dagster_dbt import dbt_assets, build_dbt_asset_selection
@dbt_assets(manifest=...)
def all_dbt_assets():
...
# Select the dbt assets that have the tag "foo".
foo_selection = build_dbt_asset_selection([dbt_assets], dbt_select="tag:foo")
# Select the dbt assets that have the tag "foo" and all Dagster assets downstream
# of them (dbt-related or otherwise)
foo_and_downstream_selection = foo_selection.downstream()
```
Building an asset selection on a dbt assets definition with an existing selection:
```python
from dagster_dbt import dbt_assets, build_dbt_asset_selection
@dbt_assets(
manifest=...
select="bar+",
)
def bar_plus_dbt_assets():
...
# Select the dbt assets that are in the intersection of having the tag "foo" and being
# in the existing selection "bar+".
bar_plus_and_foo_selection = build_dbt_asset_selection(
[bar_plus_dbt_assets],
dbt_select="tag:foo"
)
# Furthermore, select all assets downstream (dbt-related or otherwise).
bar_plus_and_foo_and_downstream_selection = bar_plus_and_foo_selection.downstream()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.build_schedule_from_dbt_selection'>dagster_dbt.build_schedule_from_dbt_selection <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L380' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.build_schedule_from_dbt_selection" class="hash-link"></a></Link></dt>
<dd>
Build a schedule to materialize a specified set of dbt resources from a dbt selection string.
See [https://docs.getdbt.com/reference/node-selection/syntax#how-does-selection-work](https://docs.getdbt.com/reference/node-selection/syntax#how-does-selection-work) for
more information.
Parameters:
- <strong>job_name</strong> (<em>str</em>) – The name of the job to materialize the dbt resources.
- <strong>cron_schedule</strong> (<em>str</em>) – The cron schedule to define the schedule.
- <strong>dbt_select</strong> (<em>str</em>) – A dbt selection string to specify a set of dbt resources.
- <strong>dbt_exclude</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A dbt selection string to exclude a set of dbt resources.
- <strong>dbt_selector</strong> (<em>str</em>) – A dbt selector to select resources to materialize.
- <strong>schedule_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the dbt schedule to create.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – A dictionary of tags (string key-value pairs) to attach to the scheduled runs.
- <strong>config</strong> (<em>Optional</em><em>[</em>[*RunConfig*](../../../api/dagster/config.mdx#dagster.RunConfig)<em>]</em>) – The config that parameterizes the execution of this schedule.
- <strong>execution_timezone</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Timezone in which the schedule should run. Supported strings for timezones are the ones provided by the <cite>IANA time zone database \<https://www.iana.org/time-zones></cite> - e.g. “America/Los_Angeles”.
Returns: A definition to materialize the selected dbt resources on a cron schedule.Return type: [ScheduleDefinition](../../../api/dagster/schedules-sensors.mdx#dagster.ScheduleDefinition)
Examples:
```python
from dagster_dbt import dbt_assets, build_schedule_from_dbt_selection
@dbt_assets(manifest=...)
def all_dbt_assets():
...
daily_dbt_assets_schedule = build_schedule_from_dbt_selection(
[all_dbt_assets],
job_name="all_dbt_assets",
cron_schedule="0 0 * * *",
dbt_select="fqn:*",
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.get_asset_key_for_model'>dagster_dbt.get_asset_key_for_model <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L142' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.get_asset_key_for_model" class="hash-link"></a></Link></dt>
<dd>
Return the corresponding Dagster asset key for a dbt model, seed, or snapshot.
Parameters:
- <strong>dbt_assets</strong> ([*AssetsDefinition*](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)) – An AssetsDefinition object produced by @dbt_assets.
- <strong>model_name</strong> (<em>str</em>) – The name of the dbt model, seed, or snapshot.
Returns: The corresponding Dagster asset key.Return type: [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)
Examples:
```python
from dagster import asset
from dagster_dbt import dbt_assets, get_asset_key_for_model
@dbt_assets(manifest=...)
def all_dbt_assets():
...
@asset(deps={get_asset_key_for_model([all_dbt_assets], "customers")})
def cleaned_customers():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.get_asset_key_for_source'>dagster_dbt.get_asset_key_for_source <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L251' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.get_asset_key_for_source" class="hash-link"></a></Link></dt>
<dd>
Returns the corresponding Dagster asset key for a dbt source with a singular table.
Parameters: <strong>source_name</strong> (<em>str</em>) – The name of the dbt source.Raises: <strong>DagsterInvalidInvocationError</strong> – If the source has more than one table.Returns: The corresponding Dagster asset key.Return type: [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)
Examples:
```python
from dagster import asset
from dagster_dbt import dbt_assets, get_asset_key_for_source
@dbt_assets(manifest=...)
def all_dbt_assets():
...
@asset(key=get_asset_key_for_source([all_dbt_assets], "my_source"))
def upstream_python_asset():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.get_asset_keys_by_output_name_for_source'>dagster_dbt.get_asset_keys_by_output_name_for_source <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L190' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.get_asset_keys_by_output_name_for_source" class="hash-link"></a></Link></dt>
<dd>
Returns the corresponding Dagster asset keys for all tables in a dbt source.
This is a convenience method that makes it easy to define a multi-asset that generates
all the tables for a given dbt source.
Parameters: <strong>source_name</strong> (<em>str</em>) – The name of the dbt source.Returns:
A mapping of the table name to corresponding Dagster asset key
for all tables in the given dbt source.
Return type: Mapping[str, [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)]
Examples:
```python
from dagster import AssetOut, multi_asset
from dagster_dbt import dbt_assets, get_asset_keys_by_output_name_for_source
@dbt_assets(manifest=...)
def all_dbt_assets():
...
@multi_asset(
outs={
name: AssetOut(key=asset_key)
for name, asset_key in get_asset_keys_by_output_name_for_source(
[all_dbt_assets], "raw_data"
).items()
},
)
def upstream_python_asset():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtProject'>`class` dagster_dbt.DbtProject <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dbt_project.py#L185' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtProject" class="hash-link"></a></Link></dt>
<dd>
Representation of a dbt project and related settings that assist with managing the project preparation.
Using this helps achieve a setup where the dbt manifest file
and dbt dependencies are available and up-to-date:
* during development, pull the dependencies and reload the manifest at run time to pick up any changes.
* when deployed, expect a manifest that was created at build time to reduce start-up time.
The cli `dagster-dbt project prepare-and-package` can be used as part of the deployment process to
handle the project preparation.
This object can be passed directly to [`DbtCliResource`](#dagster_dbt.DbtCliResource).
Parameters:
- <strong>project_dir</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em>) – The directory of the dbt project.
- <strong>target_path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em>) – The path, relative to the project directory, to output artifacts. It corresponds to the target path in dbt. Default: “target”
- <strong>profiles_dir</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em>) – The path to the directory containing your dbt <cite>profiles.yml</cite>. By default, the current working directory is used, which is the dbt project directory.
- <strong>profile</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The profile from your dbt <cite>profiles.yml</cite> to use for execution, if it should be explicitly set.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The target from your dbt <cite>profiles.yml</cite> to use for execution, if it should be explicitly set.
- <strong>packaged_project_dir</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em><em>]</em>) – A directory that will contain a copy of the dbt project and the manifest.json when the artifacts have been built. The prepare method will handle syncing the project_path to this directory. This is useful when the dbt project needs to be part of the python package data like when deploying using PEX.
- <strong>state_path</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em><em>]</em>) – The path, relative to the project directory, to reference artifacts from another run.
Examples:
Creating a DbtProject with by referencing the dbt project directory:
```python
from pathlib import Path
from dagster_dbt import DbtProject
my_project = DbtProject(project_dir=Path("path/to/dbt_project"))
```
Creating a DbtProject that changes target based on environment variables and uses manged state artifacts:
```python
import os
from pathlib import Path
from dagster_dbt import DbtProject
def get_env():
if os.getenv("DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT", "") == "1":
return "BRANCH"
if os.getenv("DAGSTER_CLOUD_DEPLOYMENT_NAME", "") == "prod":
return "PROD"
return "LOCAL"
dbt_project = DbtProject(
project_dir=Path('path/to/dbt_project'),
state_path="target/managed_state",
target=get_env(),
)
```
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtProject.prepare_if_dev'>prepare_if_dev <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/dbt_project.py#L342' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtProject.prepare_if_dev" class="hash-link"></a></Link></dt>
<dd>
Prepare a dbt project at run time during development, i.e. when <cite>dagster dev</cite> is used.
This method has no effect outside this development context.
The preparation process ensures that the dbt manifest file and dbt dependencies are available and up-to-date.
During development, it pulls the dependencies and reloads the manifest at run time to pick up any changes.
If this method returns successfully, <cite>self.manifest_path</cite> will point to a loadable manifest file.
This method causes errors if the manifest file has not been correctly created by the preparation process.
Examples:
Preparing a DbtProject during development:
```python
from pathlib import Path
from dagster import Definitions
from dagster_dbt import DbtProject
my_project = DbtProject(project_dir=Path("path/to/dbt_project"))
my_project.prepare_if_dev()
Definitions(
resources={
"dbt": DbtCliResource(project_dir=my_project),
},
...
)
```
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="asset-checks-dbt-core">
### Asset Checks (dbt Core)
<dl>
<dt><Link class="anchor" id='dagster_dbt.build_freshness_checks_from_dbt_assets'>dagster_dbt.build_freshness_checks_from_dbt_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/freshness_builder.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.build_freshness_checks_from_dbt_assets" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Create `FreshnessPolicy` objects for your dbt models by overriding `get_asset_spec` in your `DagsterDbtTranslator`, or by updating the `translation` configuration of your `DbtProjectComponent` instead..
:::
Returns a sequence of freshness checks constructed from the provided dbt assets.
Freshness checks can be configured on a per-model basis in the model schema configuration.
For assets which are not partitioned based on time, the freshness check configuration mirrors
that of the `build_last_update_freshness_checks()` function. <cite>lower_bound_delta</cite> is provided in
terms of seconds, and <cite>deadline_cron</cite> is optional.
For time-partitioned assets, the freshness check configuration mirrors that of the
`build_time_partition_freshness_checks()` function.
Below is example of configuring a non-time-partitioned dbt asset with a freshness check.
This code would be placed in the schema.yml file for the dbt model.
```YAML
models:
- name: customers
...
meta:
dagster:
freshness_check:
lower_bound_delta_seconds: 86400 # 1 day
deadline_cron: "0 0 * * *" # Optional
severity: "WARN" # Optional, defaults to "WARN"
```
Below is an example of configuring a time-partitioned dbt asset with a freshness check.
This code would be placed in the schema.yml file for the dbt model.
```yaml
models:
- name: customers
...
meta:
dagster:
freshness_check:
deadline_cron: "0 0 * * *"
severity: "WARN" # Optional, defaults to "WARN"
```
Parameters: <strong>dbt_assets</strong> (<em>Sequence</em><em>[</em>[*AssetsDefinition*](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)<em>]</em>) – A sequence of dbt assets to construct freshness
checks from.Returns:
A sequence of asset checks definitions representing the
freshness checks for the provided dbt assets.
Return type: Sequence[[AssetChecksDefinition](../../../api/dagster/asset-checks.mdx#dagster.AssetChecksDefinition)]
</dd>
</dl>
</div>
<div class="section" id="resources-dbt-core">
### Resources (dbt Core)
<div class="section" id="cli-resource">
#### CLI resource
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliResource'>`class` dagster_dbt.DbtCliResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliResource" class="hash-link"></a></Link></dt>
<dd>
A resource used to execute dbt CLI commands.
Parameters:
- <strong>project_dir</strong> (<em>str</em>) – The path to the dbt project directory. This directory should contain a <cite>dbt_project.yml</cite>. See [https://docs.getdbt.com/reference/dbt_project.yml](https://docs.getdbt.com/reference/dbt_project.yml) for more information.
- <strong>global_config_flags</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – A list of global flags configuration to pass to the dbt CLI invocation. Invoke <cite>dbt –help</cite> to see a full list of global flags.
- <strong>profiles_dir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The path to the directory containing your dbt <cite>profiles.yml</cite>. By default, the current working directory is used, which is the dbt project directory. See [https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles) for more information.
- <strong>profile</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The profile from your dbt <cite>profiles.yml</cite> to use for execution. See [https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles) for more information.
- <strong>target</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The target from your dbt <cite>profiles.yml</cite> to use for execution. See [https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles) for more information.
- <strong>dbt_executable</strong> (<em>str</em>) – The path to the dbt executable. By default, this is <cite>dbt</cite>.
- <strong>state_path</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The path, relative to the project directory, to a directory of dbt artifacts to be used with <cite>–state</cite> / <cite>–defer-state</cite>.
Examples:
Creating a dbt resource with only a reference to `project_dir`:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(project_dir="/path/to/dbt/project")
```
Creating a dbt resource with a custom `profiles_dir`:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(
project_dir="/path/to/dbt/project",
profiles_dir="/path/to/dbt/project/profiles",
)
```
Creating a dbt resource with a custom `profile` and `target`:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(
project_dir="/path/to/dbt/project",
profiles_dir="/path/to/dbt/project/profiles",
profile="jaffle_shop",
target="dev",
)
```
Creating a dbt resource with global configs, e.g. disabling colored logs with `--no-use-color`:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(
project_dir="/path/to/dbt/project",
global_config_flags=["--no-use-color"],
)
```
Creating a dbt resource with custom dbt executable path:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(
project_dir="/path/to/dbt/project",
dbt_executable="/path/to/dbt/executable",
)
```
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliResource.cli'>cli <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py#L493' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliResource.cli" class="hash-link"></a></Link></dt>
<dd>
Create a subprocess to execute a dbt CLI command.
Parameters:
- <strong>args</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – The dbt CLI command to execute.
- <strong>raise_on_error</strong> (<em>bool</em>) – Whether to raise an exception if the dbt CLI command fails.
- <strong>manifest</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>, </em><em>str</em><em>, </em><em>Path</em><em>]</em><em>]</em>) – The dbt manifest blob. If an execution context from within <cite>@dbt_assets</cite> is provided to the context argument, then the manifest provided to <cite>@dbt_assets</cite> will be used.
- <strong>dagster_dbt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDbtTranslator*](#dagster_dbt.DagsterDbtTranslator)<em>]</em>) – The translator to link dbt nodes to Dagster assets. If an execution context from within <cite>@dbt_assets</cite> is provided to the context argument, then the dagster_dbt_translator provided to <cite>@dbt_assets</cite> will be used.
- <strong>context</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em><em>]</em>) – The execution context from within <cite>@dbt_assets</cite>. If an AssetExecutionContext is passed, its underlying OpExecutionContext will be used.
- <strong>target_path</strong> (<em>Optional</em><em>[</em><em>Path</em><em>]</em>) – An explicit path to a target folder to use to store and retrieve dbt artifacts when running a dbt CLI command. If not provided, a unique target path will be generated.
Returns:
A invocation instance that can be used to retrieve the output of the
dbt CLI command.
Return type: [DbtCliInvocation](#dagster_dbt.DbtCliInvocation)
Examples:
Streaming Dagster events for dbt asset materializations and observations:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["run"], context=context).stream()
```
Retrieving a dbt artifact after streaming the Dagster events:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_run_invocation = dbt.cli(["run"], context=context)
yield from dbt_run_invocation.stream()
# Retrieve the `run_results.json` dbt artifact as a dictionary:
run_results_json = dbt_run_invocation.get_artifact("run_results.json")
# Retrieve the `run_results.json` dbt artifact as a file path:
run_results_path = dbt_run_invocation.target_path.joinpath("run_results.json")
```
Customizing the asset materialization metadata when streaming the Dagster events:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_cli_invocation = dbt.cli(["run"], context=context)
for dagster_event in dbt_cli_invocation.stream():
if isinstance(dagster_event, Output):
context.add_output_metadata(
metadata={
"my_custom_metadata": "my_custom_metadata_value",
},
output_name=dagster_event.output_name,
)
yield dagster_event
```
Suppressing exceptions from a dbt CLI command when a non-zero exit code is returned:
```python
from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
dbt_run_invocation = dbt.cli(["run"], context=context, raise_on_error=False)
if dbt_run_invocation.is_successful():
yield from dbt_run_invocation.stream()
else:
...
```
Invoking a dbt CLI command in a custom asset or op:
```python
import json
from dagster import Nothing, Out, asset, op
from dagster_dbt import DbtCliResource
@asset
def my_dbt_asset(dbt: DbtCliResource):
dbt_macro_args = {"key": "value"}
dbt.cli(["run-operation", "my-macro", json.dumps(dbt_macro_args)]).wait()
@op(out=Out(Nothing))
def my_dbt_op(dbt: DbtCliResource):
dbt_macro_args = {"key": "value"}
yield from dbt.cli(["run-operation", "my-macro", json.dumps(dbt_macro_args)]).stream()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliResource.get_defer_args'>get_defer_args <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py#L465' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliResource.get_defer_args" class="hash-link"></a></Link></dt>
<dd>
Build the defer arguments for the dbt CLI command, using the supplied state directory.
If no state directory is supplied, or the state directory does not have a manifest for.
comparison, an empty list of arguments is returned.
Returns: The defer arguments for the dbt CLI command.Return type: Sequence[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliResource.get_state_args'>get_state_args <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py#L479' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliResource.get_state_args" class="hash-link"></a></Link></dt>
<dd>
Build the state arguments for the dbt CLI command, using the supplied state directory.
If no state directory is supplied, or the state directory does not have a manifest for.
comparison, an empty list of arguments is returned.
Returns: The state arguments for the dbt CLI command.Return type: Sequence[str]
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation'>`class` dagster_dbt.DbtCliInvocation <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L76' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation" class="hash-link"></a></Link></dt>
<dd>
The representation of an invoked dbt command.
Parameters:
- <strong>process</strong> (<em>subprocess.Popen</em>) – The process running the dbt command.
- <strong>manifest</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The dbt manifest blob.
- <strong>project</strong> (<em>Optional</em><em>[</em>[*DbtProject*](#dagster_dbt.DbtProject)<em>]</em>) – The dbt project.
- <strong>project_dir</strong> (<em>Path</em>) – The path to the dbt project.
- <strong>target_path</strong> (<em>Path</em>) – The path to the dbt target folder.
- <strong>raise_on_error</strong> (<em>bool</em>) – Whether to raise an exception if the dbt command fails.
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.get_artifact'>get_artifact <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L381' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.get_artifact" class="hash-link"></a></Link></dt>
<dd>
Retrieve a dbt artifact from the target path.
See [https://docs.getdbt.com/reference/artifacts/dbt-artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts) for more information.
Parameters: <strong>artifact</strong> (<em>Union</em><em>[</em><em>Literal</em><em>[</em><em>"manifest.json"</em><em>]</em><em>, </em><em>Literal</em><em>[</em><em>"catalog.json"</em><em>]</em><em>, </em><em>Literal</em><em>[</em><em>"run_results.json"</em><em>]</em><em>, </em><em>Literal</em><em>[</em><em>"sources.json"</em><em>]</em><em>]</em>) – The name of the artifact to retrieve.Returns: The artifact as a dictionary.Return type: Dict[str, Any]
Examples:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(project_dir="/path/to/dbt/project")
dbt_cli_invocation = dbt.cli(["run"]).wait()
# Retrieve the run_results.json artifact.
run_results = dbt_cli_invocation.get_artifact("run_results.json")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.get_error'>get_error <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L243' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.get_error" class="hash-link"></a></Link></dt>
<dd>
Return an exception if the dbt CLI process failed.
Returns: An exception if the dbt CLI process failed, and None otherwise.Return type: Optional[Exception]
Examples:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(project_dir="/path/to/dbt/project")
dbt_cli_invocation = dbt.cli(["run"], raise_on_error=False)
error = dbt_cli_invocation.get_error()
if error:
logger.error(error)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.is_successful'>is_successful <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.is_successful" class="hash-link"></a></Link></dt>
<dd>
Return whether the dbt CLI process completed successfully.
Returns: True, if the dbt CLI process returns with a zero exit code, and False otherwise.Return type: bool
Examples:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(project_dir="/path/to/dbt/project")
dbt_cli_invocation = dbt.cli(["run"], raise_on_error=False)
if dbt_cli_invocation.is_successful():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.stream'>stream <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L295' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.stream" class="hash-link"></a></Link></dt>
<dd>
Stream the events from the dbt CLI process and convert them to Dagster events.
Returns:
A set of corresponding Dagster events.
In a Dagster asset definition, the following are yielded:
- Output for refables (e.g. models, seeds, snapshots.)
- AssetCheckResult for dbt test results that are enabled as asset checks.
- AssetObservation for dbt test results that are not enabled as asset checks.
In a Dagster op definition, the following are yielded:
- AssetMaterialization refables (e.g. models, seeds, snapshots.)
- AssetCheckEvaluation for dbt test results that are enabled as asset checks.
- AssetObservation for dbt test results that are not enabled as asset checks.
Return type: Iterator[Union[[Output](../../../api/dagster/ops.mdx#dagster.Output), [AssetMaterialization](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [AssetObservation](../../../api/dagster/assets.mdx#dagster.AssetObservation), [AssetCheckResult](../../../api/dagster/asset-checks.mdx#dagster.AssetCheckResult), AssetCheckEvaluation]]
Examples:
```python
from pathlib import Path
from dagster_dbt import DbtCliResource, dbt_assets
@dbt_assets(manifest=Path("target", "manifest.json"))
def my_dbt_assets(context, dbt: DbtCliResource):
yield from dbt.cli(["run"], context=context).stream()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.stream_raw_events'>stream_raw_events <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L330' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.stream_raw_events" class="hash-link"></a></Link></dt>
<dd>
Stream the events from the dbt CLI process.
Returns: An iterator of events from the dbt CLI process.Return type: Iterator[[DbtCliEventMessage](#dagster_dbt.DbtCliEventMessage)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliInvocation.wait'>wait <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_invocation.py#L200' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliInvocation.wait" class="hash-link"></a></Link></dt>
<dd>
Wait for the dbt CLI process to complete.
Returns: The current representation of the dbt CLI invocation.Return type: [DbtCliInvocation](#dagster_dbt.DbtCliInvocation)
Examples:
```python
from dagster_dbt import DbtCliResource
dbt = DbtCliResource(project_dir="/path/to/dbt/project")
dbt_cli_invocation = dbt.cli(["run"]).wait()
```
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.core.dbt_cli_invocation.DbtEventIterator'>`class` dagster_dbt.core.dbt_cli_invocation.DbtEventIterator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_event_iterator.py#L203' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator" class="hash-link"></a></Link></dt>
<dd>
A wrapper around an iterator of dbt events which contains additional methods for
post-processing the events, such as fetching row counts for materialized tables.
<dl>
<dt><Link class="anchor" id='dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_column_metadata'>fetch_column_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_event_iterator.py#L237' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_column_metadata" class="hash-link"></a></Link></dt>
<dd>
Functionality which will fetch column schema metadata for dbt models in a run
once they’re built. It will also fetch schema information for upstream models and generate
column lineage metadata using sqlglot, if enabled.
Parameters: <strong>with_column_lineage</strong> (<em>bool</em>) – Whether to generate column lineage metadata using sqlglot.Returns: A set of corresponding Dagster events for dbt models, with column metadata attached,
yielded in the order they are emitted by dbt.Return type: Iterator[Union[[Output](../../../api/dagster/ops.mdx#dagster.Output), [AssetMaterialization](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [AssetObservation](../../../api/dagster/assets.mdx#dagster.AssetObservation), [AssetCheckResult](../../../api/dagster/asset-checks.mdx#dagster.AssetCheckResult), AssetCheckEvaluation]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_row_counts'>fetch_row_counts <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_event_iterator.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_row_counts" class="hash-link"></a></Link></dt>
<dd>
Functionality which will fetch row counts for materialized dbt
models in a dbt run once they are built. Note that row counts will not be fetched
for views, since this requires running the view’s SQL query which may be costly.
Returns: A set of corresponding Dagster events for dbt models, with row counts attached,
yielded in the order they are emitted by dbt.Return type: Iterator[Union[[Output](../../../api/dagster/ops.mdx#dagster.Output), [AssetMaterialization](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [AssetObservation](../../../api/dagster/assets.mdx#dagster.AssetObservation), [AssetCheckResult](../../../api/dagster/asset-checks.mdx#dagster.AssetCheckResult), AssetCheckEvaluation]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.with_insights'>with_insights <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_event_iterator.py#L322' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.with_insights" class="hash-link"></a></Link></dt>
<dd>
Associate each warehouse query with the produced asset materializations for use in Dagster
Plus Insights. Currently supports Snowflake and BigQuery. For any other adapter (e.g.
DuckDB), this is a no-op: a warning is logged and the dbt events pass through unchanged,
so the same pipeline can run locally against an unsupported warehouse without failing.
For more information, see the documentation for
<cite>dagster_cloud.dagster_insights.dbt_with_snowflake_insights</cite> and
<cite>dagster_cloud.dagster_insights.dbt_with_bigquery_insights</cite>.
Parameters:
- <strong>skip_config_check</strong> (<em>bool</em>) – If true, skips the check that the dbt project config is set up correctly. Defaults to False.
- <strong>record_observation_usage</strong> (<em>bool</em>) – If True, associates the usage associated with asset observations with that asset. Default is True.
<strong>Example:</strong>
```python
@dbt_assets(manifest=DBT_MANIFEST_PATH)
def jaffle_shop_dbt_assets(
context: AssetExecutionContext,
dbt: DbtCliResource,
):
yield from dbt.cli(["build"], context=context).stream().with_insights()
```
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliEventMessage'>`class` dagster_dbt.DbtCliEventMessage <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_event.py#L220' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliEventMessage" class="hash-link"></a></Link></dt>
<dd>
The representation of a dbt CLI event.
Parameters:
- <strong>raw_event</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – The raw event dictionary. See [https://docs.getdbt.com/reference/events-logging#structured-logging](https://docs.getdbt.com/reference/events-logging#structured-logging) for more information.
- <strong>event_history_metadata</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary of metadata about the current event, gathered from previous historical events.
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCliEventMessage.to_default_asset_events'>to_default_asset_events <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/core/dbt_cli_event.py#L537' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCliEventMessage.to_default_asset_events" class="hash-link"></a></Link></dt>
<dd>
Convert a dbt CLI event to a set of corresponding Dagster events.
Parameters:
- <strong>manifest</strong> (<em>Union</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>, </em><em>str</em><em>, </em><em>Path</em><em>]</em>) – The dbt manifest blob.
- <strong>dagster_dbt_translator</strong> ([*DagsterDbtTranslator*](#dagster_dbt.DagsterDbtTranslator)) – Optionally, a custom translator for linking dbt nodes to Dagster assets.
- <strong>context</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em><em>]</em>) – The execution context.
- <strong>target_path</strong> (<em>Optional</em><em>[</em><em>Path</em><em>]</em>) – An explicit path to a target folder used to retrieve dbt artifacts while generating events.
Returns:
A set of corresponding Dagster events.
>
In a Dagster asset definition, the following are yielded:
- Output for refables (e.g. models, seeds, snapshots.)
- AssetCheckResult for dbt test results that are enabled as asset checks.
- AssetObservation for dbt test results that are not enabled as asset checks.
In a Dagster op definition, the following are yielded:
- AssetMaterialization refables (e.g. models, seeds, snapshots.)
- AssetCheckEvaluation for dbt test results that are enabled as asset checks.
- AssetObservation for dbt test results that are not enabled as asset checks.
Return type: Iterator[Union[[Output](../../../api/dagster/ops.mdx#dagster.Output), [AssetMaterialization](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [AssetObservation](../../../api/dagster/assets.mdx#dagster.AssetObservation), [AssetCheckResult](../../../api/dagster/asset-checks.mdx#dagster.AssetCheckResult), AssetCheckEvaluation]]
</dd>
</dl>
</dd>
</dl>
</div></div></div>
<div class="section" id="dbt-cloud-v2">
## dbt Cloud v2
Updated interfaces to manage dbt projects invoked by the hosted dbt Cloud service.
<div class="section" id="assets-dbt-cloud-v2">
### Assets (dbt Cloud v2)
<dl>
<dt><Link class="anchor" id='dagster_dbt.dbt_cloud_assets'>@dagster_dbt.dbt_cloud_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/asset_decorator.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.dbt_cloud_assets" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to compute a set of dbt Cloud resources,
described by a manifest.json for a given dbt Cloud workspace.
Parameters:
- <strong>workspace</strong> ([*DbtCloudWorkspace*](#dagster_dbt.DbtCloudWorkspace)) – The dbt Cloud workspace.
- <strong>select</strong> (<em>str</em>) – A dbt selection string for the models in a project that you want to include. Defaults to `fqn:*`.
- <strong>exclude</strong> (<em>str</em>) – A dbt selection string for the models in a project that you want to exclude. Defaults to “”.
- <strong>selector</strong> (<em>str</em>) – A dbt selector to select resources to materialize. Defaults to “”.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the op.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the asset group.
- <strong>dagster_dbt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDbtTranslator*](#dagster_dbt.DagsterDbtTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert dbt Cloud content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterDbtTranslator`](#dagster_dbt.DagsterDbtTranslator).
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the dbt Cloud assets.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](../../../api/dagster/partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – If a partitions_def is defined, this determines the behavior when launching backfills for the assets. Defaults to single-run backfill policy when a TimeWindowPartitionsDefinition is provided.
</dd>
</dl>
</div>
<div class="section" id="resources-dbt-cloud-v2">
### Resources (dbt Cloud v2)
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudCredentials'>`class` dagster_dbt.DbtCloudCredentials <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/resources.py#L61' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudCredentials" class="hash-link"></a></Link></dt>
<dd>
The DbtCloudCredentials to access your dbt Cloud workspace.
Parameters:
- <strong>account_id</strong> (<em>int</em>) – The ID of your dbt Cloud account.
- <strong>token</strong> (<em>str</em>) – Your dbt Cloud API token.
- <strong>access_url</strong> (<em>str</em>) – Your dbt Cloud workspace URL.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudWorkspace'>`class` dagster_dbt.DbtCloudWorkspace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/resources.py#L76' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudWorkspace" class="hash-link"></a></Link></dt>
<dd>
This class represents a dbt Cloud workspace and provides utilities
to interact with dbt Cloud APIs.
Parameters:
- <strong>credentials</strong> ([*DbtCloudCredentials*](#dagster_dbt.DbtCloudCredentials)) – An instance of DbtCloudCredentials class.
- <strong>project_id</strong> (<em>int</em>) – The ID of the dbt cloud project to use for this resource.
- <strong>environment_id</strong> (<em>int</em>) – The ID of the environment to use for the dbt Cloud project used in this resource.
- <strong>adhoc_job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the ad hoc job that will be created by Dagster in your dbt Cloud workspace. This ad hoc job is used to parse your project and materialize your dbt Cloud assets. If not provided, this job name will be generated using your project ID and environment ID. When `adhoc_job_pool_size > 1`, this value is used as a prefix and the additional jobs receive an `__\{index}` suffix.
- <strong>adhoc_job_pool_size</strong> (<em>int</em>) – The number of ad hoc jobs to create in your dbt Cloud workspace. dbt Cloud only allows one concurrent run per job, so a value greater than 1 lets Dagster run multiple dbt Cloud invocations concurrently (e.g., partitioned backfills, two Dagster jobs targeting different assets). Defaults to 1.
- <strong>adhoc_job_pool_mode</strong> (<em>Literal</em><em>[</em><em>"overflow"</em><em>, </em><em>"wait"</em><em>, </em><em>"fail"</em><em>]</em>) – What to do when every ad hoc job in the pool already has an active run at `cli()` time. `overflow` (default) triggers the run on the first job regardless and lets dbt Cloud queue it. `wait` polls until a job frees up. `fail` raises immediately.
- <strong>request_max_retries</strong> (<em>int</em>) – The maximum number of times requests to the dbt Cloud API should be retried before failing.
- <strong>request_retry_delay</strong> (<em>float</em>) – Time (in seconds) to wait between each request retry.
- <strong>request_timeout</strong> – Time (in seconds) after which the requests to dbt Cloud are declared timed out.
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudWorkspace.cli'>cli <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud_v2/resources.py#L374' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudWorkspace.cli" class="hash-link"></a></Link></dt>
<dd>
Creates a dbt CLI invocation with the dbt Cloud client.
Parameters:
- <strong>args</strong> – (Sequence[str]): The dbt CLI command to execute.
- <strong>dagster_dbt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDbtTranslator*](#dagster_dbt.DagsterDbtTranslator)<em>]</em>) – Allows customizing how to map dbt models, seeds, etc. to asset keys and asset metadata.
- <strong>context</strong> (<em>Optional</em><em>[</em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The execution context.
</dd>
</dl>
</dd>
</dl>
</div></div>
<div class="section" id="dbt-cloud">
## dbt Cloud
Here, we provide interfaces to manage dbt projects invoked by the hosted dbt Cloud service.
<div class="section" id="assets-dbt-cloud">
### Assets (dbt Cloud)
<dl>
<dt><Link class="anchor" id='dagster_dbt.load_assets_from_dbt_cloud_job'>dagster_dbt.load_assets_from_dbt_cloud_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud/asset_defs.py#L545' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.load_assets_from_dbt_cloud_job" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Loads a set of dbt models, managed by a dbt Cloud job, into Dagster assets. In order to
determine the set of dbt models, the project is compiled to generate the necessary artifacts
that define the dbt models and their dependencies.
One Dagster asset is created for each dbt model.
Parameters:
- <strong>dbt_cloud</strong> ([*ResourceDefinition*](../../../api/dagster/resources.mdx#dagster.ResourceDefinition)) – The dbt Cloud resource to use to connect to the dbt Cloud API.
- <strong>job_id</strong> (<em>int</em>) – The ID of the dbt Cloud job to load assets from.
- <strong>node_info_to_asset_key</strong> – (Mapping[str, Any] -> AssetKey): A function that takes a dictionary of dbt metadata and returns the AssetKey that you want to represent a given model or source. By default: dbt model -> AssetKey([model_name]) and dbt source -> AssetKey([source_name, table_name])
- <strong>node_info_to_group_fn</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>] </em><em>-> Optional</em><em>[</em><em>str</em><em>]</em>) – A function that takes a dictionary of dbt node info and returns the group that this node should be assigned to.
- <strong>node_info_to_auto_materialize_policy_fn</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>] </em><em>-> Optional</em><em>[</em><em>AutoMaterializePolicy</em><em>]</em>) – A function that takes a dictionary of dbt node info and optionally returns a AutoMaterializePolicy that should be applied to this node. By default, AutoMaterializePolicies will be created from config applied to dbt models, i.e.: <cite>dagster_auto_materialize_policy=\{“type”: “lazy”}</cite> will result in that model being assigned <cite>AutoMaterializePolicy.lazy()</cite>
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – <span className="flag flag-info">beta</span> Defines the set of partition keys that compose the dbt assets.
- <strong>partition_key_to_vars_fn</strong> (<em>Optional</em><em>[</em><em>str -> Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> A function to translate a given partition key (e.g. ‘2022-01-01’) to a dictionary of vars to be passed into the dbt invocation (e.g. \{“run_date”: “2022-01-01”})
Returns: A definition for the loaded assets.Return type: CacheableAssetsDefinition
Examples:
```python
from dagster import repository
from dagster_dbt import dbt_cloud_resource, load_assets_from_dbt_cloud_job
DBT_CLOUD_JOB_ID = 1234
dbt_cloud = dbt_cloud_resource.configured(
{
"auth_token": {"env": "DBT_CLOUD_API_TOKEN"},
"account_id": {"env": "DBT_CLOUD_ACCOUNT_ID"},
}
)
dbt_cloud_assets = load_assets_from_dbt_cloud_job(
dbt_cloud=dbt_cloud, job_id=DBT_CLOUD_JOB_ID
)
@repository
def dbt_cloud_sandbox():
return [dbt_cloud_assets]
```
</dd>
</dl>
</div>
<div class="section" id="ops-dbt-cloud">
### Ops (dbt Cloud)
<dl>
<dt><Link class="anchor" id='dagster_dbt.dbt_cloud_run_op'>dagster_dbt.dbt_cloud_run_op `=` \<dagster._core.definitions.op_definition.OpDefinition object> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud/ops.py#L48' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.dbt_cloud_run_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Initiates a run for a dbt Cloud job, then polls until the run completes. If the job
fails or is otherwised stopped before succeeding, a <cite>dagster.Failure</cite> exception will be raised,
and this op will fail.
It requires the use of a ‘dbt_cloud’ resource, which is used to connect to the dbt Cloud API.
<strong>Config Options:</strong>
job_id (int)
The integer ID of the relevant dbt Cloud job. You can find this value by going to the details
page of your job in the dbt Cloud UI. It will be the final number in the url, e.g.:
`https://cloud.getdbt.com/#/accounts/\{account_id}/projects/\{project_id}/jobs/\{job_id}/`
poll_interval (float)
The time (in seconds) that will be waited between successive polls. Defaults to `10`.
poll_timeout (float)
The maximum time (in seconds) that will waited before this operation is timed out. By
default, this will never time out.
yield_materializations (bool)
If True, materializations corresponding to the results of the dbt operation will be
yielded when the solid executes. Defaults to `True`.
rasset_key_prefix (float)
If provided and yield_materializations is True, these components will be used to ”
prefix the generated asset keys. Defaults to [“dbt”].
<strong>Examples:</strong>
```python
from dagster import job
from dagster_dbt import dbt_cloud_resource, dbt_cloud_run_op
my_dbt_cloud_resource = dbt_cloud_resource.configured(
{"auth_token": {"env": "DBT_CLOUD_AUTH_TOKEN"}, "account_id": 77777}
)
run_dbt_nightly_sync = dbt_cloud_run_op.configured(
{"job_id": 54321}, name="run_dbt_nightly_sync"
)
@job(resource_defs={"dbt_cloud": my_dbt_cloud_resource})
def dbt_cloud():
run_dbt_nightly_sync()
```
</dd>
</dl>
</div>
<div class="section" id="resources-dbt-cloud">
### Resources (dbt Cloud)
<dl>
<dt><Link class="anchor" id='dagster_dbt.DbtCloudClientResource'>`class` dagster_dbt.DbtCloudClientResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud/resources.py#L595' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DbtCloudClientResource" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource helps interact with dbt Cloud connectors.
</dd>
</dl>
<div class="section" id="deprecated-dbt-cloud">
#### Deprecated (dbt Cloud)
<dl>
<dt><Link class="anchor" id='dagster_dbt.dbt_cloud_resource'>dagster_dbt.dbt_cloud_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/cloud/resources.py#L661' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.dbt_cloud_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource allows users to programatically interface with the dbt Cloud Administrative REST
API (v2) to launch jobs and monitor their progress. This currently implements only a subset of
the functionality exposed by the API.
For a complete set of documentation on the dbt Cloud Administrative REST API, including expected
response JSON schemae, see the [dbt Cloud API Docs](https://docs.getdbt.com/dbt-cloud/api-v2).
To configure this resource, we recommend using the [configured](https://legacy-docs.dagster.io/concepts/configuration/configured) method.
<strong>Examples:</strong>
```python
from dagster import job
from dagster_dbt import dbt_cloud_resource
my_dbt_cloud_resource = dbt_cloud_resource.configured(
{
"auth_token": {"env": "DBT_CLOUD_AUTH_TOKEN"},
"account_id": {"env": "DBT_CLOUD_ACCOUNT_ID"},
}
)
@job(resource_defs={"dbt_cloud": my_dbt_cloud_resource})
def my_dbt_cloud_job():
...
```
</dd>
</dl>
</div></div></div>
<div class="section" id="errors">
## Errors
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtError'>`exception` dagster_dbt.DagsterDbtError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/errors.py#L6' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtError" class="hash-link"></a></Link></dt>
<dd>
The base exception of the `dagster-dbt` library.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.DagsterDbtCliRuntimeError'>`exception` dagster_dbt.DagsterDbtCliRuntimeError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/errors.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.DagsterDbtCliRuntimeError" class="hash-link"></a></Link></dt>
<dd>
Represents an error while executing a dbt CLI command.
</dd>
</dl>
</div>
<div class="section" id="utils">
## Utils
<dl>
<dt><Link class="anchor" id='dagster_dbt.default_group_from_dbt_resource_props'>dagster_dbt.default_group_from_dbt_resource_props <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L712' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.default_group_from_dbt_resource_props" class="hash-link"></a></Link></dt>
<dd>
Get the group name for a dbt node.
If a Dagster group is configured in the metadata for the node, use that.
Otherwise, if a dbt group is configured for the node, use that.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.group_from_dbt_resource_props_fallback_to_directory'>dagster_dbt.group_from_dbt_resource_props_fallback_to_directory <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L732' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.group_from_dbt_resource_props_fallback_to_directory" class="hash-link"></a></Link></dt>
<dd>
Get the group name for a dbt node.
Has the same behavior as the default_group_from_dbt_resource_props, except for that, if no group can be determined
from config or metadata, falls back to using the subdirectory of the models directory that the
source file is in.
Parameters: <strong>dbt_resource_props</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the dbt resource.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dbt.default_metadata_from_dbt_resource_props'>dagster_dbt.default_metadata_from_dbt_resource_props <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/asset_utils.py#L670' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dbt.default_metadata_from_dbt_resource_props" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div></div>
---
---
title: 'dagster-deltalake-pandas library'
sidebar_position: 1000
title_meta: 'dagster-deltalake-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-deltalake-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-deltalake-pandas-library">
# dagster-deltalake-pandas library
This library provides an integration with the [Delta Lake](https://delta.io) storage framework.
Related guides:
- [Using Dagster with Delta Lake guide](https://docs.dagster.io/integrations/libraries/deltalake)
- [DeltaLake I/O manager reference](https://docs.dagster.io/integrations/libraries/deltalake/reference)
<dl>
<dt><Link class="anchor" id='dagster_deltalake_pandas.DeltaLakePandasIOManager'>dagster_deltalake_pandas.DeltaLakePandasIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake-pandas/dagster_deltalake_pandas/deltalake_pandas_type_handler.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake_pandas.DeltaLakePandasIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to Delta Lake.
Examples:
```python
from dagster_deltalake import DeltaLakeIOManager
from dagster_deltalake_pandas import DeltaLakePandasTypeHandler
class MyDeltaLakeIOManager(DeltaLakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DeltaLakePandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema (parent folder) in Delta Lake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDeltaLakeIOManager()}
)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O Manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_deltalake_pandas.DeltaLakePandasTypeHandler'>`class` dagster_deltalake_pandas.DeltaLakePandasTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake-pandas/dagster_deltalake_pandas/deltalake_pandas_type_handler.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake_pandas.DeltaLakePandasTypeHandler" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
---
---
title: 'dagster-deltalake-polars library'
sidebar_position: 1000
title_meta: 'dagster-deltalake-polars library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-deltalake-polars library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-deltalake-polars-library">
# dagster-deltalake-polars library
This library provides an integration with the [Delta Lake](https://delta.io) storage framework.
Related guides:
- [Using Dagster with Delta Lake guide](https://docs.dagster.io/integrations/libraries/deltalake)
- [DeltaLake I/O manager reference](https://docs.dagster.io/integrations/libraries/deltalake/reference)
<dl>
<dt><Link class="anchor" id='dagster_deltalake_polars.DeltaLakePolarsIOManager'>dagster_deltalake_polars.DeltaLakePolarsIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake-polars/dagster_deltalake_polars/deltalake_polars_type_handler.py#L66' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake_polars.DeltaLakePolarsIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to Delta Lake.
Examples:
```python
from dagster_deltalake import DeltaLakeIOManager
from dagster_deltalake_pandas import DeltaLakePandasTypeHandler
class MyDeltaLakeIOManager(DeltaLakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DeltaLakePandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema (parent folder) in Delta Lake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDeltaLakeIOManager()}
)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O Manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_deltalake_polars.DeltaLakePolarsTypeHandler'>`class` dagster_deltalake_polars.DeltaLakePolarsTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake-polars/dagster_deltalake_polars/deltalake_polars_type_handler.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake_polars.DeltaLakePolarsTypeHandler" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
---
---
title: 'dagster-deltalake library'
sidebar_position: 1000
title_meta: 'dagster-deltalake library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-deltalake library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-deltalake-library">
# dagster-deltalake library
This library provides an integration with the [Delta Lake](https://delta.io) storage framework.
Related Guides:
- [Using Dagster with Delta Lake tutorial](https://docs.dagster.io/integrations/libraries/deltalake)
- [Delta Lake reference](https://docs.dagster.io/integrations/libraries/deltalake/reference)
<dl>
<dt><Link class="anchor" id='dagster_deltalake.DeltaLakeIOManager'>dagster_deltalake.DeltaLakeIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake/dagster_deltalake/io_manager.py#L71' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake.DeltaLakeIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to Delta Lake.
Examples:
```python
from dagster_deltalake import DeltaLakeIOManager
from dagster_deltalake_pandas import DeltaLakePandasTypeHandler
class MyDeltaLakeIOManager(DeltaLakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DeltaLakePandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema (parent folder) in Delta Lake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDeltaLakeIOManager()}
)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O Manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_deltalake.DeltaLakePyarrowIOManager'>dagster_deltalake.DeltaLakePyarrowIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake/dagster_deltalake/__init__.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake.DeltaLakePyarrowIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to Delta Lake.
Examples:
```python
from dagster_deltalake import DeltaLakeIOManager
from dagster_deltalake_pandas import DeltaLakePandasTypeHandler
class MyDeltaLakeIOManager(DeltaLakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DeltaLakePandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema (parent folder) in Delta Lake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDeltaLakeIOManager()}
)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O Manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_deltalake.DeltaTableResource'>dagster_deltalake.DeltaTableResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-deltalake/dagster_deltalake/resource.py#L8' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_deltalake.DeltaTableResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with a Delta table.
Examples:
```python
from dagster import Definitions, asset
from dagster_deltalake import DeltaTableResource, LocalConfig
@asset
def my_table(delta_table: DeltaTableResource):
df = delta_table.load().to_pandas()
Definitions(
assets=[my_table],
resources={
"delta_table": DeltaTableResource(
url="/path/to/table",
storage_options=LocalConfig()
)
}
)
```
</dd>
</dl>
</div>
---
---
title: 'dagster-dlt library'
sidebar_position: 1000
title_meta: 'dagster-dlt library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-dlt library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-dlt-library">
# dagster-dlt library
This library provides a Dagster integration with [dlt](https://dlthub.com).
For more information on getting started, see the [Dagster & dlt](https://docs.dagster.io/integrations/libraries/dlt) documentation.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_dlt.DltLoadCollectionComponent'>`class` dagster_dlt.DltLoadCollectionComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/components/dlt_load_collection/component.py#L110' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DltLoadCollectionComponent" class="hash-link"></a></Link></dt>
<dd>
Expose one or more dlt loads to Dagster as assets.
<dl>
<dt><Link class="anchor" id='dagster_dlt.DltLoadCollectionComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/components/dlt_load_collection/component.py#L184' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DltLoadCollectionComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes the dlt pipeline for the selected resources.
This method can be overridden in a subclass to customize the pipeline execution behavior,
such as adding custom logging, validation, or error handling.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>dlt_pipeline_resource</strong> – The DagsterDltResource used to run the dlt pipeline
Yields: Events from the dlt pipeline execution (e.g., AssetMaterialization, MaterializeResult)
Example:
Override this method to add custom logging during pipeline execution:
```python
from dagster_dlt import DltLoadCollectionComponent
from dagster import AssetExecutionContext
class CustomDltLoadCollectionComponent(DltLoadCollectionComponent):
def execute(self, context, dlt_pipeline_resource):
context.log.info("Starting dlt pipeline execution")
yield from super().execute(context, dlt_pipeline_resource)
context.log.info("dlt pipeline execution completed")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DltLoadCollectionComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/components/dlt_load_collection/component.py#L128' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DltLoadCollectionComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given dlt resource.
This method can be overridden in a subclass to customize how dlt resources are
converted to Dagster asset specs. By default, it delegates to the configured
DagsterDltTranslator.
Parameters: <strong>data</strong> – The DltResourceTranslatorData containing information about the dlt source
and resource being loadedReturns: An AssetSpec that represents the dlt resource as a Dagster asset
Example:
Override this method to add custom tags based on resource properties:
```python
from dagster_dlt import DltLoadCollectionComponent
from dagster import AssetSpec
class CustomDltLoadCollectionComponent(DltLoadCollectionComponent):
def get_asset_spec(self, data):
base_spec = super().get_asset_spec(data)
return base_spec.replace_attributes(
tags={
**base_spec.tags,
"source": data.source_name,
"resource": data.resource_name
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the dlt component, see the [dlt component integration guide](https://docs.dagster.io/integrations/libraries/dlt).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a dlt component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_dlt.DltLoadCollectionComponent
attributes:
loads:
- source: .loads.my_load_source
pipeline: .loads.my_load_pipeline
```
</div></div>
<div class="section" id="assets">
## Assets
<dl>
<dt><Link class="anchor" id='dagster_dlt.dlt_assets'>@dagster_dlt.dlt_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/asset_decorator.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.dlt_assets" class="hash-link"></a></Link></dt>
<dd>
Asset Factory for using data load tool (dlt).
Parameters:
- <strong>dlt_source</strong> (<em>DltSource</em>) – The DltSource to be ingested.
- <strong>dlt_pipeline</strong> (<em>Pipeline</em>) – The dlt Pipeline defining the destination parameters.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the op.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the asset group.
- <strong>dagster_dlt_translator</strong> ([*DagsterDltTranslator*](#dagster_dlt.DagsterDltTranslator)<em>, </em><em>optional</em>) – Customization object for defining asset parameters from dlt resources.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Optional partitions definition.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](../../../api/dagster/partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – If a partitions_def is defined, this determines how to execute backfills that target multiple partitions. If a time window partition definition is used, this parameter defaults to a single-run policy.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The tags for the underlying op.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs the dlt assets’ execution.
Examples:
Loading Hubspot data to Snowflake with an auto materialize policy using the dlt verified source:
```python
from dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assets
class HubspotDagsterDltTranslator(DagsterDltTranslator):
@public
def get_auto_materialize_policy(self, resource: DltResource) -> Optional[AutoMaterializePolicy]:
return AutoMaterializePolicy.eager().with_rules(
AutoMaterializeRule.materialize_on_cron("0 0 * * *")
)
@dlt_assets(
dlt_source=hubspot(include_history=True),
dlt_pipeline=pipeline(
pipeline_name="hubspot",
dataset_name="hubspot",
destination="snowflake",
progress="log",
),
name="hubspot",
group_name="hubspot",
dagster_dlt_translator=HubspotDagsterDltTranslator(),
)
def hubspot_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
yield from dlt.run(context=context)
```
Loading Github issues to snowflake:
```python
from dagster_dlt import DagsterDltResource, dlt_assets
@dlt_assets(
dlt_source=github_reactions(
"dagster-io", "dagster", items_per_page=100, max_items=250
),
dlt_pipeline=pipeline(
pipeline_name="github_issues",
dataset_name="github",
destination="snowflake",
progress="log",
),
name="github",
group_name="github",
)
def github_reactions_dagster_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
yield from dlt.run(context=context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.build_dlt_asset_specs'>dagster_dlt.build_dlt_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/asset_decorator.py#L22' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.build_dlt_asset_specs" class="hash-link"></a></Link></dt>
<dd>
Build a list of asset specs from a dlt source and pipeline.
Parameters:
- <strong>dlt_source</strong> (<em>DltSource</em>) – dlt source object
- <strong>dlt_pipeline</strong> (<em>Pipeline</em>) – dlt pipeline object
- <strong>dagster_dlt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDltTranslator*](#dagster_dlt.DagsterDltTranslator)<em>]</em>) – Allows customizing how to map dlt project to asset keys and asset metadata.
Returns: List[AssetSpec] list of asset specs from dlt source and pipeline
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator'>`class` dagster_dlt.DagsterDltTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_asset_key'>get_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L95' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_asset_key" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).key` instead..
:::
Defines asset key for a given dlt resource key and dataset name.
This method can be overridden to provide custom asset key for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: AssetKey of Dagster asset derived from dlt resource
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_auto_materialize_policy'>get_auto_materialize_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L125' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_auto_materialize_policy" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).auto_materialize_policy` instead..
:::
Defines resource specific auto materialize policy.
This method can be overridden to provide custom auto materialize policy for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: The auto-materialize policy for a resourceReturn type: Optional[AutoMaterializePolicy]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_automation_condition'>get_automation_condition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L157' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_automation_condition" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).automation_condition` instead..
:::
Defines resource specific automation condition.
This method can be overridden to provide custom automation condition for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: The automation condition for a resourceReturn type: Optional[[AutomationCondition](../../../api/dagster/assets.mdx#dagster.AutomationCondition)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_deps_asset_keys'>get_deps_asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L190' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_deps_asset_keys" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Iterate over `DagsterDltTranslator.get_asset_spec(...).deps` to access `AssetDep.asset_key` instead..
:::
Defines upstream asset dependencies given a dlt resource.
Defaults to a concatenation of <cite>resource.source_name</cite> and <cite>resource.name</cite>.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: The Dagster asset keys upstream of <cite>dlt_resource_key</cite>.Return type: Iterable[[AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_description'>get_description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L227' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_description" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).description` instead..
:::
A method that takes in a dlt resource returns the Dagster description of the resource.
This method can be overridden to provide a custom description for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: The Dagster description for the dlt resource.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_group_name'>get_group_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L260' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_group_name" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).group_name` instead..
:::
A method that takes in a dlt resource and returns the Dagster group name of the resource.
This method can be overridden to provide a custom group name for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: A Dagster group name for the dlt resource.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_kinds'>get_kinds <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L375' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_kinds" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).kinds` instead..
:::
A method that takes in a dlt resource and returns the kinds which should be
attached. Defaults to the destination type and “dlt”.
This method can be overridden to provide custom kinds for a dlt resource.
Parameters:
- <strong>resource</strong> (<em>DltResource</em>) – dlt resource
- <strong>destination</strong> (<em>Destination</em>) – dlt destination
Returns: The kinds of the asset.Return type: Set[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_metadata'>get_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L288' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_metadata" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).metadata` instead..
:::
Defines resource specific metadata.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: The custom metadata entries for this resource.Return type: Mapping[str, Any]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_owners'>get_owners <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L317' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_owners" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).owners` instead..
:::
A method that takes in a dlt resource and returns the Dagster owners of the resource.
This method can be overridden to provide custom owners for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns: A sequence of Dagster owners for the dlt resource.Return type: Optional[Sequence[str]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltTranslator.get_tags'>get_tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/translator.py#L345' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltTranslator.get_tags" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterDltTranslator.get_asset_spec(...).tags` instead..
:::
A method that takes in a dlt resource and returns the Dagster tags of the structure.
This method can be overridden to provide custom tags for a dlt resource.
Parameters: <strong>resource</strong> (<em>DltResource</em>) – dlt resourceReturns:
A dictionary representing the Dagster tags for the
dlt resource.
Return type: Optional[Mapping[str, str]]
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltResource'>`class` dagster_dlt.DagsterDltResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/resource.py#L29' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltResource" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_dlt.DagsterDltResource.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dlt/dagster_dlt/resource.py#L177' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_dlt.DagsterDltResource.run" class="hash-link"></a></Link></dt>
<dd>
Runs the dlt pipeline with subset support.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – Asset or op execution context
- <strong>dlt_source</strong> (<em>Optional</em><em>[</em><em>DltSource</em><em>]</em>) – optional dlt source if resource is used from an <cite>@op</cite>
- <strong>dlt_pipeline</strong> (<em>Optional</em><em>[</em><em>Pipeline</em><em>]</em>) – optional dlt pipeline if resource is used from an <cite>@op</cite>
- <strong>dagster_dlt_translator</strong> (<em>Optional</em><em>[</em>[*DagsterDltTranslator*](#dagster_dlt.DagsterDltTranslator)<em>]</em>) – optional dlt translator if resource is used from an <cite>@op</cite>
- <strong>**kwargs</strong> (<em>dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – Keyword args passed to pipeline <cite>run</cite> method
Returns: An iterator of MaterializeResult or AssetMaterializationReturn type: DltEventIterator[DltEventType]
</dd>
</dl>
</dd>
</dl>
</div></div>
---
---
title: 'dagster-docker library'
sidebar_position: 1000
title_meta: 'dagster-docker library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-docker library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-docker-library">
# dagster-docker library
<div class="section" id="apis">
## APIs
<dl>
<dt><Link class="anchor" id='dagster_docker.DockerRunLauncher'>dagster_docker.DockerRunLauncher RunLauncher<a href="#dagster_docker.DockerRunLauncher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Launches runs in a Docker container.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_docker.docker_executor'>dagster_docker.docker_executor ExecutorDefinition<a href="#dagster_docker.docker_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Executor which launches steps as Docker containers.
To use the <cite>docker_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster_docker import docker_executor
from dagster import job
@job(executor_def=docker_executor)
def docker_job():
pass
```
Then you can configure the executor with run config as follows:
```YAML
execution:
config:
registry: ...
network: ...
networks: ...
container_kwargs: ...
```
If you’re using the DockerRunLauncher, configuration set on the containers created by the run
launcher will also be set on the containers that are created for each step.
</dd>
</dl>
<div class="section" id="ops">
### Ops
<dl>
<dt><Link class="anchor" id='dagster_docker.docker_container_op'>dagster_docker.docker_container_op `=` \<dagster._core.definitions.op_definition.OpDefinition object><a href="#dagster_docker.docker_container_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
An op that runs a Docker container using the docker Python API.
Contrast with the <cite>docker_executor</cite>, which runs each Dagster op in a Dagster job in its
own Docker container.
This op may be useful when:
- You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)
- You want to run the rest of a Dagster job using a specific executor, and only a single op in docker.
For example:
```python
from dagster_docker import docker_container_op
from dagster import job
first_op = docker_container_op.configured(
{
"image": "busybox",
"command": ["echo HELLO"],
},
name="first_op",
)
second_op = docker_container_op.configured(
{
"image": "busybox",
"command": ["echo GOODBYE"],
},
name="second_op",
)
@job
def full_job():
second_op(first_op())
```
You can create your own op with the same implementation by calling the <cite>execute_docker_container</cite> function
inside your own op.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_docker.execute_docker_container'>dagster_docker.execute_docker_container<a href="#dagster_docker.execute_docker_container" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This function is a utility for executing a Docker container from within a Dagster op.
Parameters:
- <strong>image</strong> (<em>str</em>) – The image to use for the launched Docker container.
- <strong>entrypoint</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The ENTRYPOINT to run in the launched Docker container. Default: None.
- <strong>command</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The CMD to run in the launched Docker container. Default: None.
- <strong>networks</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Names of the Docker networks to which to connect the launched container. Default: None.
- <strong>registry</strong> – (Optional[Mapping[str, str]]): Information for using a non local/public Docker registry. Can have “url”, “username”, or “password” keys.
- <strong>env_vars</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) – List of environemnt variables to include in the launched container. ach can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the calling environment.
- <strong>container_kwargs</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>[</em><em>Any</em><em>]</em><em>]</em><em>]</em>) – key-value pairs that can be passed into containers.create in the Docker Python API. See [https://docker-py.readthedocs.io/en/stable/containers.html](https://docker-py.readthedocs.io/en/stable/containers.html) for the full list of available options.
</dd>
</dl>
</div>
<div class="section" id="pipes">
### Pipes
<dl>
<dt><Link class="anchor" id='dagster_docker.PipesDockerClient'>`class` dagster_docker.PipesDockerClient<a href="#dagster_docker.PipesDockerClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client that runs external processes in docker containers.
By default context is injected via environment variables and messages are parsed out of the
log stream, with other logs forwarded to stdout of the orchestration process.
Parameters:
- <strong>env</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – An optional dict of environment variables to pass to the container.
- <strong>register</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – An optional dict of registry credentials to login to the docker client.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the docker container process. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the docker container process. Defaults to `DockerLogsMessageReader`.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_docker.PipesDockerLogsMessageReader'>`class` dagster_docker.PipesDockerLogsMessageReader<a href="#dagster_docker.PipesDockerLogsMessageReader" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-fivetran library'
sidebar_position: 1000
title_meta: 'dagster-fivetran library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-fivetran library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-fivetran-library">
# dagster-fivetran library
This library provides a Dagster integration with [Fivetran](https://www.fivetran.com).
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranAccountComponent'>`class` dagster_fivetran.FivetranAccountComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/components/workspace_component/component.py#L82' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranAccountComponent" class="hash-link"></a></Link></dt>
<dd>
Loads Fivetran connectors from a given Fivetran instance as Dagster assets.
Materializing these assets will trigger a sync of the Fivetran connector, enabling
you to schedule Fivetran syncs using Dagster.
Example:
```yaml
# defs.yaml
type: dagster_fivetran.FivetranAccountComponent
attributes:
workspace:
account_id: your_account_id
api_key: "{{ env.FIVETRAN_API_KEY }}"
api_secret: "{{ env.FIVETRAN_API_SECRET }}"
connector_selector:
by_name:
- my_postgres_connector
- my_snowflake_connector
```
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranAccountComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/components/workspace_component/component.py#L202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranAccountComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes a Fivetran sync for the selected connector.
This method can be overridden in a subclass to customize the sync execution behavior,
such as adding custom logging or handling sync results differently.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>fivetran</strong> – The FivetranWorkspace resource used to trigger and monitor syncs
Yields: AssetMaterialization or MaterializeResult events from the Fivetran sync
Example:
Override this method to add custom logging during sync execution:
```python
from dagster_fivetran import FivetranAccountComponent
import dagster as dg
class CustomFivetranAccountComponent(FivetranAccountComponent):
def execute(self, context, fivetran):
context.log.info("Starting Fivetran sync")
yield from super().execute(context, fivetran)
context.log.info("Fivetran sync completed successfully")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranAccountComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/components/workspace_component/component.py#L166' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranAccountComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Fivetran connector table.
This method can be overridden in a subclass to customize how Fivetran connector tables
are converted to Dagster asset specs. By default, it delegates to the configured
DagsterFivetranTranslator.
Parameters: <strong>props</strong> – The FivetranConnectorTableProps containing information about the connector
and destination table being syncedReturns: An AssetSpec that represents the Fivetran connector table as a Dagster asset
Example:
Override this method to add custom tags based on connector properties:
```python
from dagster_fivetran import FivetranAccountComponent
import dagster as dg
class CustomFivetranAccountComponent(FivetranAccountComponent):
def get_asset_spec(self, props):
base_spec = super().get_asset_spec(props)
return base_spec.replace_attributes(
tags={
**base_spec.tags,
"connector_type": props.connector_type,
"destination": props.destination_name
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Fivetran component, see the [Fivetran component integration guide](https://docs.dagster.io/integrations/libraries/fivetran).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Fivetran component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_fivetran.FivetranAccountComponent
attributes:
workspace:
account_id: test_account
api_key: '{{ env.FIVETRAN_API_KEY }}'
api_secret: '{{ env.FIVETRAN_API_SECRET }}'
```
</div></div>
<div class="section" id="assets-fivetran-api">
## Assets (Fivetran API)
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranWorkspace'>`class` dagster_fivetran.FivetranWorkspace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py#L996' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranWorkspace" class="hash-link"></a></Link></dt>
<dd>
This class represents a Fivetran workspace and provides utilities
to interact with Fivetran APIs.
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranWorkspace.sync_and_poll'>sync_and_poll <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py#L1277' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranWorkspace.sync_and_poll" class="hash-link"></a></Link></dt>
<dd>
Executes a sync and poll process to materialize Fivetran assets.
This method can only be used in the context of an asset execution.
Parameters:
- <strong>context</strong> ([*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)) – The execution context from within <cite>@fivetran_assets</cite>.
- <strong>config</strong> (<em>Optional</em><em>[</em><em>FivetranSyncConfig</em><em>]</em>) – Optional configuration to control sync behavior. If config.resync is True, performs a historical resync instead of a normal sync. If config.resync_parameters is provided, only the specified tables will be resynced.
Returns:
An iterator of MaterializeResult
or AssetMaterialization.
Return type: Iterator[Union[[AssetMaterialization](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [MaterializeResult](../../../api/dagster/assets.mdx#dagster.MaterializeResult)]]
Examples:
Normal sync (without config):
```python
from dagster import AssetExecutionContext
from dagster_fivetran import FivetranWorkspace, fivetran_assets
@fivetran_assets(connector_id="my_connector", workspace=fivetran_workspace)
def my_fivetran_assets(context: AssetExecutionContext, fivetran: FivetranWorkspace):
yield from fivetran.sync_and_poll(context=context)
```
Historical resync of specific tables (config passed at runtime):
```python
from dagster import AssetExecutionContext
from dagster_fivetran import FivetranWorkspace, FivetranSyncConfig, fivetran_assets
@fivetran_assets(connector_id="my_connector", workspace=fivetran_workspace)
def my_fivetran_assets(
context: AssetExecutionContext,
fivetran: FivetranWorkspace,
config: FivetranSyncConfig,
):
# When materializing, pass config with:
# resync=True
# resync_parameters={"schema_name": ["table1", "table2"]}
yield from fivetran.sync_and_poll(context=context, config=config)
```
Full historical resync (config passed at runtime):
```python
from dagster import AssetExecutionContext
from dagster_fivetran import FivetranWorkspace, FivetranSyncConfig, fivetran_assets
@fivetran_assets(connector_id="my_connector", workspace=fivetran_workspace)
def my_fivetran_assets(
context: AssetExecutionContext,
fivetran: FivetranWorkspace,
config: FivetranSyncConfig,
):
# When materializing, pass config with resync=True to resync all tables
yield from fivetran.sync_and_poll(context=context, config=config)
```
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.DagsterFivetranTranslator'>`class` dagster_fivetran.DagsterFivetranTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/translator.py#L355' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.DagsterFivetranTranslator" class="hash-link"></a></Link></dt>
<dd>
Translator class which converts a <cite>FivetranConnectorTableProps</cite> object into AssetSpecs.
Subclass this class to implement custom logic on how to translate Fivetran content into asset spec.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.fivetran_assets'>@dagster_fivetran.fivetran_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_decorator.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.fivetran_assets" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to sync the tables of a given Fivetran connector.
Parameters:
- <strong>connector_id</strong> (<em>str</em>) – The Fivetran Connector ID. You can retrieve this value from the “Setup” tab of a given connector in the Fivetran UI.
- <strong>workspace</strong> ([*FivetranWorkspace*](#dagster_fivetran.FivetranWorkspace)) – The Fivetran workspace to fetch assets from.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the op.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the asset group.
- <strong>dagster_fivetran_translator</strong> (<em>Optional</em><em>[</em>[*DagsterFivetranTranslator*](#dagster_fivetran.DagsterFivetranTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Fivetran content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterFivetranTranslator`](#dagster_fivetran.DagsterFivetranTranslator).
- <strong>connector_selector_fn</strong> (<em>Optional</em><em>[</em><em>ConnectorSelectorFn</em><em>]</em>) – A function that allows for filtering which Fivetran connector assets are created for.
Examples:
Sync the tables of a Fivetran connector:
```python
from dagster_fivetran import FivetranWorkspace, fivetran_assets
import dagster as dg
fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)
@fivetran_assets(
connector_id="fivetran_connector_id",
name="fivetran_connector_id",
group_name="fivetran_connector_id",
workspace=fivetran_workspace,
)
def fivetran_connector_assets(context: dg.AssetExecutionContext, fivetran: FivetranWorkspace):
yield from fivetran.sync_and_poll(context=context)
defs = dg.Definitions(
assets=[fivetran_connector_assets],
resources={"fivetran": fivetran_workspace},
)
```
Sync the tables of a Fivetran connector with a custom translator:
```python
from dagster_fivetran import (
DagsterFivetranTranslator,
FivetranConnectorTableProps,
FivetranWorkspace,
fivetran_assets
)
import dagster as dg
class CustomDagsterFivetranTranslator(DagsterFivetranTranslator):
def get_asset_spec(self, props: FivetranConnectorTableProps) -> dg.AssetSpec:
default_spec = super().get_asset_spec(props)
return default_spec.replace_attributes(
key=default_spec.key.with_prefix("my_prefix"),
)
fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)
@fivetran_assets(
connector_id="fivetran_connector_id",
name="fivetran_connector_id",
group_name="fivetran_connector_id",
workspace=fivetran_workspace,
dagster_fivetran_translator=CustomDagsterFivetranTranslator(),
)
def fivetran_connector_assets(context: dg.AssetExecutionContext, fivetran: FivetranWorkspace):
yield from fivetran.sync_and_poll(context=context)
defs = dg.Definitions(
assets=[fivetran_connector_assets],
resources={"fivetran": fivetran_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.load_fivetran_asset_specs'>dagster_fivetran.load_fivetran_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py#L1450' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.load_fivetran_asset_specs" class="hash-link"></a></Link></dt>
<dd>
Returns a list of AssetSpecs representing the Fivetran content in the workspace.
Parameters:
- <strong>workspace</strong> ([*FivetranWorkspace*](#dagster_fivetran.FivetranWorkspace)) – The Fivetran workspace to fetch assets from.
- <strong>dagster_fivetran_translator</strong> (<em>Optional</em><em>[</em>[*DagsterFivetranTranslator*](#dagster_fivetran.DagsterFivetranTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Fivetran content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterFivetranTranslator`](#dagster_fivetran.DagsterFivetranTranslator).
- <strong>connector_selector_fn</strong> (<em>Optional</em><em>[</em><em>ConnectorSelectorFn</em><em>]</em>) – A function that allows for filtering which Fivetran connector assets are created for.
Returns: The set of assets representing the Fivetran content in the workspace.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
Examples:
Loading the asset specs for a given Fivetran workspace:
```python
from dagster_fivetran import FivetranWorkspace, load_fivetran_asset_specs
import dagster as dg
fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)
fivetran_specs = load_fivetran_asset_specs(fivetran_workspace)
defs = dg.Definitions(assets=[*fivetran_specs], resources={"fivetran": fivetran_workspace}
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.build_fivetran_assets_definitions'>dagster_fivetran.build_fivetran_assets_definitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py#L716' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.build_fivetran_assets_definitions" class="hash-link"></a></Link></dt>
<dd>
The list of AssetsDefinition for all connectors in the Fivetran workspace.
Parameters:
- <strong>workspace</strong> ([*FivetranWorkspace*](#dagster_fivetran.FivetranWorkspace)) – The Fivetran workspace to fetch assets from.
- <strong>dagster_fivetran_translator</strong> (<em>Optional</em><em>[</em>[*DagsterFivetranTranslator*](#dagster_fivetran.DagsterFivetranTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Fivetran content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterFivetranTranslator`](#dagster_fivetran.DagsterFivetranTranslator).
- <strong>connector_selector_fn</strong> (<em>Optional</em><em>[</em><em>ConnectorSelectorFn</em><em>]</em>) – A function that allows for filtering which Fivetran connector assets are created for.
Returns: The list of AssetsDefinition for all connectors in the Fivetran workspace.Return type: List[[AssetsDefinition](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)]
Examples:
Sync the tables of a Fivetran connector:
```python
from dagster_fivetran import FivetranWorkspace, build_fivetran_assets_definitions
import dagster as dg
fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)
fivetran_assets = build_fivetran_assets_definitions(workspace=workspace)
defs = dg.Definitions(
assets=[*fivetran_assets],
resources={"fivetran": fivetran_workspace},
)
```
Sync the tables of a Fivetran connector with a custom translator:
```python
from dagster_fivetran import (
DagsterFivetranTranslator,
FivetranConnectorTableProps,
FivetranWorkspace,
build_fivetran_assets_definitions
)
import dagster as dg
class CustomDagsterFivetranTranslator(DagsterFivetranTranslator):
def get_asset_spec(self, props: FivetranConnectorTableProps) -> dg.AssetSpec:
default_spec = super().get_asset_spec(props)
return default_spec.replace_attributes(
key=default_spec.key.with_prefix("my_prefix"),
)
fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)
fivetran_assets = build_fivetran_assets_definitions(
workspace=workspace,
dagster_fivetran_translator=CustomDagsterFivetranTranslator()
)
defs = dg.Definitions(
assets=[*fivetran_assets],
resources={"fivetran": fivetran_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.fivetran_event_iterator.FivetranEventIterator'>`class` dagster_fivetran.fivetran_event_iterator.FivetranEventIterator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/fivetran_event_iterator.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.fivetran_event_iterator.FivetranEventIterator" class="hash-link"></a></Link></dt>
<dd>
A wrapper around an iterator of Fivetran events which contains additional methods for
post-processing the events, such as fetching column metadata.
<dl>
<dt><Link class="anchor" id='dagster_fivetran.fivetran_event_iterator.FivetranEventIterator.fetch_column_metadata'>fetch_column_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/fivetran_event_iterator.py#L90' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.fivetran_event_iterator.FivetranEventIterator.fetch_column_metadata" class="hash-link"></a></Link></dt>
<dd>
Fetches column metadata for each table synced with the Fivetran API.
Retrieves the column schema for each destination table.
Returns: An iterator of Dagster events with column metadata attached.Return type: [FivetranEventIterator](#dagster_fivetran.fivetran_event_iterator.FivetranEventIterator)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.ConnectorSelectorFn'>dagster_fivetran.ConnectorSelectorFn<a href="#dagster_fivetran.ConnectorSelectorFn" class="hash-link"></a></Link></dt>
<dd>
alias of `Callable`[[FivetranConnector], `bool`]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.build_fivetran_polling_sensor'>dagster_fivetran.build_fivetran_polling_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/sensor_builder.py#L255' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.build_fivetran_polling_sensor" class="hash-link"></a></Link></dt>
<dd>
Creates a sensor that polls a Fivetran workspace for externally-triggered sync completions
and emits AssetMaterialization events into Dagster’s event log.
This is useful when Fivetran connectors are run on Fivetran’s auto-schedule (not triggered
by Dagster) and you want Dagster to be aware of the resulting table updates.
Parameters:
- <strong>workspace</strong> ([*FivetranWorkspace*](#dagster_fivetran.FivetranWorkspace)) – The Fivetran workspace to poll.
- <strong>dagster_fivetran_translator</strong> (<em>Optional</em><em>[</em>[*DagsterFivetranTranslator*](#dagster_fivetran.DagsterFivetranTranslator)<em>]</em>) – The translator to use to convert Fivetran content into AssetSpec. Defaults to DagsterFivetranTranslator.
- <strong>connector_selector_fn</strong> (<em>Optional</em><em>[</em><em>ConnectorSelectorFn</em><em>]</em>) – A function to filter which connectors are polled. If None, all connectors are polled.
- <strong>minimum_interval_seconds</strong> (<em>int</em>) – The minimum interval in seconds between sensor runs. Defaults to 30.
- <strong>default_sensor_status</strong> (<em>Optional</em><em>[</em><em>DefaultSensorStatus</em><em>]</em>) – The default status of the sensor.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the sensor. Defaults to `fivetran_\{account_id}__sync_status_sensor`.
Returns: A sensor definition.Return type: [SensorDefinition](../../../api/dagster/schedules-sensors.mdx#dagster.SensorDefinition)
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_fivetran.fivetran_resource'>dagster_fivetran.fivetran_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py#L451' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.fivetran_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 0.30.
Use `FivetranWorkspace` instead..
:::
This resource allows users to programatically interface with the Fivetran REST API to launch
syncs and monitor their progress. This currently implements only a subset of the functionality
exposed by the API.
For a complete set of documentation on the Fivetran REST API, including expected response JSON
schemae, see the [Fivetran API Docs](https://fivetran.com/docs/rest-api/connectors).
To configure this resource, we recommend using the [configured](https://legacy-docs.dagster.io/concepts/configuration/configured) method.
<strong>Examples:</strong>
```python
from dagster import job
from dagster_fivetran import fivetran_resource
my_fivetran_resource = fivetran_resource.configured(
{
"api_key": {"env": "FIVETRAN_API_KEY"},
"api_secret": {"env": "FIVETRAN_API_SECRET"},
}
)
@job(resource_defs={"fivetran":my_fivetran_resource})
def my_fivetran_job():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.FivetranResource'>dagster_fivetran.FivetranResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py#L84' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.FivetranResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 0.30.
Use `FivetranWorkspace` instead..
:::
This class exposes methods on top of the Fivetran REST API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.load_assets_from_fivetran_instance'>dagster_fivetran.load_assets_from_fivetran_instance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py#L590' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.load_assets_from_fivetran_instance" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 0.30.
Use the `build_fivetran_assets_definitions` factory instead..
:::
Loads Fivetran connector assets from a configured FivetranResource instance. This fetches information
about defined connectors at initialization time, and will error on workspace load if the Fivetran
instance is not reachable.
Parameters:
- <strong>fivetran</strong> ([*ResourceDefinition*](../../../api/dagster/resources.mdx#dagster.ResourceDefinition)) – A FivetranResource configured with the appropriate connection details.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>CoercibleToAssetKeyPrefix</em><em>]</em>) – A prefix for the asset keys created.
- <strong>connector_to_group_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Function which returns an asset group name for a given Fivetran connector name. If None, no groups will be created. Defaults to a basic sanitization function.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The IO manager key to use for all assets. Defaults to “io_manager”. Use this if all assets should be loaded from the same source, otherwise use connector_to_io_manager_key_fn.
- <strong>connector_to_io_manager_key_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>str</em><em>]</em><em>, </em><em>Optional</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Function which returns an IO manager key for a given Fivetran connector name. When other ops are downstream of the loaded assets, the IOManager specified determines how the inputs to those ops are loaded. Defaults to “io_manager”.
- <strong>connector_filter</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>FivetranConnectorMetadata</em><em>]</em><em>, </em><em>bool</em><em>]</em><em>]</em>) – Optional function which takes in connector metadata and returns False if the connector should be excluded from the output assets.
- <strong>connector_to_asset_key_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>FivetranConnectorMetadata</em><em>, </em><em>str</em><em>]</em><em>, </em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>]</em><em>]</em>) – Optional function which takes in connector metadata and a table name and returns an AssetKey for that table. Defaults to a function that generates an AssetKey matching the table name, split by “.”.
- <strong>destination_ids</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of destination IDs to fetch connectors from. If None, all destinations will be polled for connectors.
- <strong>poll_interval</strong> (<em>float</em>) – The time (in seconds) that will be waited between successive polls.
- <strong>poll_timeout</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – The maximum time that will waited before this operation is timed out. By default, this will never time out.
- <strong>fetch_column_metadata</strong> (<em>bool</em>) – If True, will fetch column schema information for each table in the connector. This will induce additional API calls.
<strong>Examples:</strong>
Loading all Fivetran connectors as assets:
```python
from dagster_fivetran import fivetran_resource, load_assets_from_fivetran_instance
fivetran_instance = fivetran_resource.configured(
{
"api_key": "some_key",
"api_secret": "some_secret",
}
)
fivetran_assets = load_assets_from_fivetran_instance(fivetran_instance)
```
Filtering the set of loaded connectors:
```python
from dagster_fivetran import fivetran_resource, load_assets_from_fivetran_instance
fivetran_instance = fivetran_resource.configured(
{
"api_key": "some_key",
"api_secret": "some_secret",
}
)
fivetran_assets = load_assets_from_fivetran_instance(
fivetran_instance,
connector_filter=lambda meta: "snowflake" in meta.name,
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.build_fivetran_assets'>dagster_fivetran.build_fivetran_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py#L226' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.build_fivetran_assets" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 0.30.
Use the `fivetran_assets` decorator instead..
:::
Build a set of assets for a given Fivetran connector.
Returns an AssetsDefinition which connects the specified `asset_keys` to the computation that
will update them. Internally, executes a Fivetran sync for a given `connector_id`, and
polls until that sync completes, raising an error if it is unsuccessful. Requires the use of the
[`fivetran_resource`](#dagster_fivetran.fivetran_resource), which allows it to communicate with the
Fivetran API.
Parameters:
- <strong>connector_id</strong> (<em>str</em>) – The Fivetran Connector ID that this op will sync. You can retrieve this value from the “Setup” tab of a given connector in the Fivetran UI.
- <strong>destination_tables</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – <cite>schema_name.table_name</cite> for each table that you want to be represented in the Dagster asset graph for this connection.
- <strong>poll_interval</strong> (<em>float</em>) – The time (in seconds) that will be waited between successive polls.
- <strong>poll_timeout</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – The maximum time that will waited before this operation is timed out. By default, this will never time out.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The io_manager to be used to handle each of these assets.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A prefix for the asset keys inside this asset. If left blank, assets will have a key of <cite>AssetKey([schema_name, table_name])</cite>.
- <strong>metadata_by_table_name</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>RawMetadataMapping</em><em>]</em><em>]</em>) – A mapping from destination table name to user-supplied metadata that should be associated with the asset for that table.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. This group name will be applied to all assets produced by this multi_asset.
- <strong>infer_missing_tables</strong> (<em>bool</em>) – If True, will create asset materializations for tables specified in destination_tables even if they are not present in the Fivetran sync output. This is useful in cases where Fivetran does not sync any data for a table and therefore does not include it in the sync output API response.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that json.loads(json.dumps(value)) == value.
- <strong>fetch_column_metadata</strong> (<em>bool</em>) – If True, will fetch column schema information for each table in the connector. This will induce additional API calls.
<strong>Examples:</strong>
Basic example:
>
```python
from dagster import AssetKey, repository, with_resources
from dagster_fivetran import fivetran_resource
from dagster_fivetran.assets import build_fivetran_assets
my_fivetran_resource = fivetran_resource.configured(
{
"api_key": {"env": "FIVETRAN_API_KEY"},
"api_secret": {"env": "FIVETRAN_API_SECRET"},
}
)
```
Attaching metadata:
>
```python
fivetran_assets = build_fivetran_assets(
connector_id="foobar",
table_names=["schema1.table1", "schema2.table2"],
metadata_by_table_name={
"schema1.table1": {
"description": "This is a table that contains foo and bar",
},
"schema2.table2": {
"description": "This is a table that contains baz and quux",
},
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_fivetran.fivetran_sync_op'>dagster_fivetran.fivetran_sync_op `=` \<dagster._core.definitions.op_definition.OpDefinition object> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/ops.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_fivetran.fivetran_sync_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 0.30.
Fivetran ops are no longer best practice and will soon be removed. Use `FivetranWorkspace` resource and `@fivetran_asset` decorator instead..
:::
Executes a Fivetran sync for a given `connector_id`, and polls until that sync
completes, raising an error if it is unsuccessful. It outputs a FivetranOutput which contains
the details of the Fivetran connector after the sync successfully completes, as well as details
about which tables the sync updates.
It requires the use of the [`fivetran_resource`](#dagster_fivetran.fivetran_resource), which allows it to
communicate with the Fivetran API.
Examples:
```python
from dagster import job
from dagster_fivetran import fivetran_resource, fivetran_sync_op
my_fivetran_resource = fivetran_resource.configured(
{
"api_key": {"env": "FIVETRAN_API_KEY"},
"api_secret": {"env": "FIVETRAN_API_SECRET"},
}
)
sync_foobar = fivetran_sync_op.configured({"connector_id": "foobar"}, name="sync_foobar")
@job(resource_defs={"fivetran": my_fivetran_resource})
def my_simple_fivetran_job():
sync_foobar()
@job(resource_defs={"fivetran": my_fivetran_resource})
def my_composed_fivetran_job():
final_foobar_state = sync_foobar(start_after=some_op())
other_op(final_foobar_state)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-gcp-pandas library'
sidebar_position: 1000
title_meta: 'dagster-gcp-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-gcp-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-gcp-pandas-library">
# dagster-gcp-pandas library
<div class="section" id="google-bigquery">
## Google BigQuery
This library provides an integration with the [BigQuery](https://cloud.google.com/bigquery) database and Pandas data processing library.
Related Guides:
- [Using Dagster with BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery)
- [BigQuery I/O manager reference](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference)
<dl>
<dt><Link class="anchor" id='dagster_gcp_pandas.BigQueryPandasIOManager'>dagster_gcp_pandas.BigQueryPandasIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py#L207' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pandas.BigQueryPandasIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes pandas DataFrames to BigQuery.
Returns: IOManagerDefinition
Examples:
```python
from dagster_gcp_pandas import BigQueryPandasIOManager
from dagster import Definitions, EnvVar
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": BigQueryPandasIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": BigQueryPandasIOManager(project=EnvVar("GCP_PROJECT"), dataset="my_dataset")
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the “gcp_credentials” configuration.
Dagster will store this key in a temporary file and set GOOGLE_APPLICATION_CREDENTIALS to point to the file.
After the run completes, the file will be deleted, and GOOGLE_APPLICATION_CREDENTIALS will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded key with this shell command: cat $GOOGLE_APPLICATION_CREDENTIALS | base64
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp_pandas.BigQueryPandasTypeHandler'>`class` dagster_gcp_pandas.BigQueryPandasTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pandas/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pandas.BigQueryPandasTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Plugin for the BigQuery I/O Manager that can store and load Pandas DataFrames as BigQuery tables.
Examples:
```python
from dagster_gcp import BigQueryIOManager
from dagster_bigquery_pandas import BigQueryPandasTypeHandler
from dagster import Definitions, EnvVar
class MyBigQueryIOManager(BigQueryIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [BigQueryPandasTypeHandler()]
@asset(
key_prefix=["my_dataset"], # my_dataset will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": MyBigQueryIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_gcp_pandas.bigquery_pandas_io_manager'>dagster_gcp_pandas.bigquery_pandas_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/io_manager.py#L146' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pandas.bigquery_pandas_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes pandas DataFrames to BigQuery.
Returns: IOManagerDefinition
Examples:
```python
from dagster_gcp_pandas import bigquery_pandas_io_manager
from dagster import Definitions
@asset(
key_prefix=["my_dataset"], # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": bigquery_pandas_io_manager.configured({
"project": {"env": "GCP_PROJECT"}
})
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": bigquery_pandas_io_manager.configured({
"project": {"env": "GCP_PROJECT"},
"dataset": "my_dataset"
})
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the “gcp_credentials” configuration.
Dagster will store this key in a temporary file and set GOOGLE_APPLICATION_CREDENTIALS to point to the file.
After the run completes, the file will be deleted, and GOOGLE_APPLICATION_CREDENTIALS will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded key with this shell command: cat $GOOGLE_APPLICATION_CREDENTIALS | base64
</dd>
</dl>
</div></div>
---
---
title: 'dagster-gcp-pyspark library'
sidebar_position: 1000
title_meta: 'dagster-gcp-pyspark library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-gcp-pyspark library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-gcp-pyspark-library">
# dagster-gcp-pyspark library
<div class="section" id="google-bigquery">
## Google BigQuery
This library provides an integration with the [BigQuery](https://cloud.google.com/bigquery) database and PySpark data processing library.
Related Guides:
- [Using Dagster with BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery)
- [BigQuery I/O manager reference](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference)
<dl>
<dt><Link class="anchor" id='dagster_gcp_pyspark.BigQueryPySparkIOManager'>dagster_gcp_pyspark.BigQueryPySparkIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py#L206' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pyspark.BigQueryPySparkIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to BigQuery.
Returns: IOManagerDefinition
Examples:
```python
from dagster_gcp_pyspark import BigQueryPySparkIOManager
from dagster import Definitions, EnvVar
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pyspark.sql.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": BigQueryPySparkIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": BigQueryPySparkIOManager(project=EnvVar("GCP_PROJECT"), dataset="my_dataset")
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pyspark.sql.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pyspark.sql.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pyspark.sql.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the “gcp_credentials” configuration.
Dagster will store this key in a temporary file and set GOOGLE_APPLICATION_CREDENTIALS to point to the file.
After the run completes, the file will be deleted, and GOOGLE_APPLICATION_CREDENTIALS will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded key with this shell command: cat $GOOGLE_APPLICATION_CREDENTIALS | base64
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp_pyspark.BigQueryPySparkTypeHandler'>`class` dagster_gcp_pyspark.BigQueryPySparkTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp-pyspark/dagster_gcp_pyspark/bigquery/bigquery_pyspark_type_handler.py#L31' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pyspark.BigQueryPySparkTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Plugin for the BigQuery I/O Manager that can store and load PySpark DataFrames as BigQuery tables.
Examples:
```python
from dagster_gcp import BigQueryIOManager
from dagster_bigquery_pandas import BigQueryPySparkTypeHandler
from dagster import Definitions, EnvVar
class MyBigQueryIOManager(BigQueryIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [BigQueryPySparkTypeHandler()]
@asset(
key_prefix=["my_dataset"], # my_dataset will be used as the dataset in BigQuery
)
def my_table() -> pyspark.sql.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": MyBigQueryIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_gcp_pyspark.bigquery_pyspark_io_manager'>dagster_gcp_pyspark.bigquery_pyspark_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/io_manager.py#L146' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp_pyspark.bigquery_pyspark_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to BigQuery.
Returns: IOManagerDefinition
Examples:
```python
from dagster_gcp_pyspark import bigquery_pyspark_io_manager
from dagster import Definitions
@asset(
key_prefix=["my_dataset"], # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": bigquery_pyspark_io_manager.configured({
"project": {"env": "GCP_PROJECT"}
})
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": bigquery_pyspark_io_manager.configured({
"project": {"env": "GCP_PROJECT"},
"dataset": "my_dataset"
})
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pyspark.sql.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pyspark.sql.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pyspark.sql.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the “gcp_credentials” configuration.
Dagster will store this key in a temporary file and set GOOGLE_APPLICATION_CREDENTIALS to point to the file.
After the run completes, the file will be deleted, and GOOGLE_APPLICATION_CREDENTIALS will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded key with this shell command: cat $GOOGLE_APPLICATION_CREDENTIALS | base64
</dd>
</dl>
</div></div>
---
---
title: 'dagster-gcp library'
sidebar_position: 1000
title_meta: 'dagster-gcp library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-gcp library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-gcp-library">
# dagster-gcp library
<div class="section" id="bigquery">
## BigQuery
Related guides:
- [Using Dagster with BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery)
- [BigQuery I/O manager reference](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference)
<div class="section" id="bigquery-resource">
### BigQuery resource
<dl>
<dt><Link class="anchor" id='dagster_gcp.BigQueryResource'>dagster_gcp.BigQueryResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/resources.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.BigQueryResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with Google BigQuery.
Examples:
```python
from dagster import Definitions, asset
from dagster_gcp import BigQueryResource
@asset
def my_table(bigquery: BigQueryResource):
with bigquery.get_client() as client:
client.query("SELECT * FROM my_dataset.my_table")
defs = Definitions(
assets=[my_table],
resources={
"bigquery": BigQueryResource(project="my-project")
}
)
```
</dd>
</dl>
</div>
<div class="section" id="bigquery-i-o-manager">
### BigQuery I/O manager
<dl>
<dt><Link class="anchor" id='dagster_gcp.BigQueryIOManager'>dagster_gcp.BigQueryIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/io_manager.py#L189' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.BigQueryIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an I/O manager definition that reads inputs from and writes outputs to BigQuery.
Examples:
```python
from dagster_gcp import BigQueryIOManager
from dagster_bigquery_pandas import BigQueryPandasTypeHandler
from dagster import Definitions, EnvVar
class MyBigQueryIOManager(BigQueryIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [BigQueryPandasTypeHandler()]
@asset(
key_prefix=["my_dataset"] # my_dataset will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": MyBigQueryIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
defs = Definitions(
assets=[my_table],
resources={
"io_manager": MyBigQueryIOManager(project=EnvVar("GCP_PROJECT"), dataset="my_dataset")
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata `columns` to the
[`In`](../../../api/dagster/ops.mdx#dagster.In) or [`AssetIn`](../../../api/dagster/assets.mdx#dagster.AssetIn).
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the `gcp_credentials` configuration.
Dagster will store this key in a temporary file and set `GOOGLE_APPLICATION_CREDENTIALS` to point to the file.
After the run completes, the file will be deleted, and `GOOGLE_APPLICATION_CREDENTIALS` will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded with this shell command: `cat $GOOGLE_APPLICATION_CREDENTIALS | base64`
To change the write mode (default is “truncate”), you can set the `write_mode` configuration.
Supported modes: “truncate”, “replace”, “append”.
```python
defs = Definitions(
assets=[my_table],
resources={
"io_manager": BigQueryIOManager(
project=EnvVar("GCP_PROJECT"),
write_mode="replace"
)
}
)
```
</dd>
</dl>
</div>
<div class="section" id="bigquery-ops">
### BigQuery ops
<dl>
<dt><Link class="anchor" id='dagster_gcp.bq_create_dataset'>dagster_gcp.bq_create_dataset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L147' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.bq_create_dataset" class="hash-link"></a></Link></dt>
<dd>
BigQuery Create Dataset.
This op encapsulates creating a BigQuery dataset.
Expects a BQ client to be provisioned in resources as context.resources.bigquery.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.bq_delete_dataset'>dagster_gcp.bq_delete_dataset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L164' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.bq_delete_dataset" class="hash-link"></a></Link></dt>
<dd>
BigQuery Delete Dataset.
This op encapsulates deleting a BigQuery dataset.
Expects a BQ client to be provisioned in resources as context.resources.bigquery.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.bq_op_for_queries'>dagster_gcp.bq_op_for_queries <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L43' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.bq_op_for_queries" class="hash-link"></a></Link></dt>
<dd>
Executes BigQuery SQL queries.
Expects a BQ client to be provisioned in resources as context.resources.bigquery.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.import_df_to_bq'>dagster_gcp.import_df_to_bq <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L99' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.import_df_to_bq" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.import_file_to_bq'>dagster_gcp.import_file_to_bq <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L109' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.import_file_to_bq" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.import_gcs_paths_to_bq'>dagster_gcp.import_gcs_paths_to_bq <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/ops.py#L89' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.import_gcs_paths_to_bq" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
<div class="section" id="data-freshness">
### Data freshness
<dl>
<dt><Link class="anchor" id='dagster_gcp.fetch_last_updated_timestamps'>dagster_gcp.fetch_last_updated_timestamps <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/resources.py#L108' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.fetch_last_updated_timestamps" class="hash-link"></a></Link></dt>
<dd>
Get the last updated timestamps of a list BigQuery table.
Note that this only works on BigQuery tables, and not views.
Parameters:
- <strong>client</strong> (<em>bigquery.Client</em>) – The BigQuery client.
- <strong>dataset_id</strong> (<em>str</em>) – The BigQuery dataset ID.
- <strong>table_ids</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – The table IDs to get the last updated timestamp for.
Returns: A mapping of table IDs to their last updated timestamps (UTC).Return type: Mapping[str, datetime]
</dd>
</dl>
</div>
<div class="section" id="other">
### Other
<dl>
<dt><Link class="anchor" id='dagster_gcp.BigQueryError'>`class` dagster_gcp.BigQueryError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/types.py#L154' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.BigQueryError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div></div>
<div class="section" id="gcs">
## GCS
<div class="section" id="gcs-resource">
### GCS resource
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSResource'>dagster_gcp.GCSResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/resources.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with Google Cloud Storage.
Example:
```python
@asset
def my_asset(gcs: GCSResource):
client = gcs.get_client()
# client is a google.cloud.storage.Client
...
```
</dd>
</dl>
</div>
<div class="section" id="gcs-i-o-manager">
### GCS I/O manager
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSPickleIOManager'>dagster_gcp.GCSPickleIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/io_manager.py#L80' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSPickleIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using GCS for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for GCS and the backing bucket.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at `\<base_dir>/\<asset_key>`. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of `/my/base/path`, an asset with key
`AssetKey(["one", "two", "three"])` would be stored in a file called `three` in a directory
with path `/my/base/path/one/two/`.
Example usage:
1. Attach this IO manager to a set of assets.
```python
from dagster import asset, Definitions
from dagster_gcp.gcs import GCSPickleIOManager, GCSResource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": GCSPickleIOManager(
gcs_bucket="my-cool-bucket",
gcs_prefix="my-cool-prefix",
gcs=GCSResource(project="my-cool-project")
),
}
)
```
2. Attach this IO manager to your job to make it available to your ops.
```python
from dagster import job
from dagster_gcp.gcs import GCSPickleIOManager, GCSResource
@job(
resource_defs={
"io_manager": GCSPickleIOManager(
gcs=GCSResource(project="my-cool-project")
gcs_bucket="my-cool-bucket",
gcs_prefix="my-cool-prefix"
),
}
)
def my_job():
...
```
</dd>
</dl>
</div>
<div class="section" id="gcs-sensor">
### GCS sensor
<dl>
<dt><Link class="anchor" id='dagster_gcp.gcs.sensor.get_gcs_keys'>dagster_gcp.gcs.sensor.get_gcs_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/sensor.py#L7' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.gcs.sensor.get_gcs_keys" class="hash-link"></a></Link></dt>
<dd>
Return a list of updated keys in a GCS bucket.
Parameters:
- <strong>bucket</strong> (<em>str</em>) – The name of the GCS bucket.
- <strong>prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The prefix to filter the keys by.
- <strong>since_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The key to start from. If provided, only keys updated after this key will be returned.
- <strong>gcs_session</strong> (<em>Optional</em><em>[</em><em>google.cloud.storage.client.Client</em><em>]</em>) – A GCS client session. If not provided, a new session will be created.
Returns: A list of keys in the bucket, sorted by update time, that are newer than the <cite>since_key</cite>.Return type: List[str]
Example:
```python
@resource
def google_cloud_storage_client(context):
return storage.Client().from_service_account_json("my-service-account.json")
@sensor(job=my_job, required_resource_keys={"google_cloud_storage_client"})
def my_gcs_sensor(context):
since_key = context.cursor or None
new_gcs_keys = get_gcs_keys(
"my-bucket",
prefix="data",
since_key=since_key,
gcs_session=context.resources.google_cloud_storage_client
)
if not new_gcs_keys:
return SkipReason("No new gcs files found for bucket 'my-bucket'.")
for gcs_key in new_gcs_keys:
yield RunRequest(run_key=gcs_key, run_config={
"ops": {
"gcs_files": {
"config": {
"gcs_key": gcs_key
}
}
}
})
last_key = new_gcs_keys[-1]
context.update_cursor(last_key)
```
</dd>
</dl>
</div>
<div class="section" id="file-manager">
### File manager
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSFileHandle'>`class` dagster_gcp.GCSFileHandle <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/file_manager.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSFileHandle" class="hash-link"></a></Link></dt>
<dd>
A reference to a file on GCS.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSFileManagerResource'>dagster_gcp.GCSFileManagerResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/resources.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSFileManagerResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
FileManager that provides abstract access to GCS.
</dd>
</dl>
</div>
<div class="section" id="gcs-compute-log-manager">
### GCS compute log manager
<dl>
<dt><Link class="anchor" id='dagster_gcp.gcs.GCSComputeLogManager'>`class` dagster_gcp.gcs.GCSComputeLogManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/compute_log_manager.py#L30' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.gcs.GCSComputeLogManager" class="hash-link"></a></Link></dt>
<dd>
Logs op compute function stdout and stderr to GCS.
Users should not instantiate this class directly. Instead, use a YAML block in `dagster.yaml`
such as the following:
```YAML
compute_logs:
module: dagster_gcp.gcs.compute_log_manager
class: GCSComputeLogManager
config:
bucket: "mycorp-dagster-compute-logs"
local_dir: "/tmp/cool"
prefix: "dagster-test-"
upload_interval: 30
```
There are more configuration examples in the instance documentation guide: [https://docs.dagster.io/deployment/oss/oss-instance-configuration#compute-log-storage](https://docs.dagster.io/deployment/oss/oss-instance-configuration#compute-log-storage)
Parameters:
- <strong>bucket</strong> (<em>str</em>) – The name of the GCS bucket to which to log.
- <strong>local_dir</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Path to the local directory in which to stage logs. Default: `dagster_shared.seven.get_system_temp_directory()`.
- <strong>prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Prefix for the log file keys.
- <strong>json_credentials_envvar</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Environment variable that contains the JSON with a private key and other credentials information. If this is set, `GOOGLE_APPLICATION_CREDENTIALS` will be ignored. Can be used when the private key cannot be used as a file.
- <strong>upload_interval</strong> – (Optional[int]): Interval in seconds to upload partial log files to GCS. By default, will only upload when the capture is complete.
- <strong>show_url_only</strong> – (Optional[bool]): Only show the URL of the log file in the UI, instead of fetching and displaying the full content. Default False.
- <strong>inst_data</strong> (<em>Optional</em><em>[</em>[*ConfigurableClassData*](../../../api/dagster/internals.mdx#dagster._serdes.ConfigurableClassData)<em>]</em>) – Serializable representation of the compute log manager when instantiated from config.
</dd>
</dl>
</div></div>
<div class="section" id="dataproc">
## Dataproc
<div class="section" id="dataproc-resource">
### Dataproc resource
<dl>
<dt><Link class="anchor" id='dagster_gcp.DataprocResource'>dagster_gcp.DataprocResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/dataproc/resources.py#L158' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.DataprocResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Resource for connecting to a Dataproc cluster.
Example:
```default
@asset
def my_asset(dataproc: DataprocResource):
with dataproc.get_client() as client:
# client is a dagster_gcp.DataprocClient
...
```
</dd>
</dl>
</div>
<div class="section" id="dataproc-ops">
### Dataproc ops
<dl>
<dt><Link class="anchor" id='dagster_gcp.dataproc_op'>dagster_gcp.dataproc_op `=` \<dagster._core.definitions.op_definition.OpDefinition object> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/dataproc/ops.py#L96' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.dataproc_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
</dd>
</dl>
</div></div>
<div class="section" id="pipes">
## Pipes
<div class="section" id="clients">
### Clients
<dl>
<dt><Link class="anchor" id='dagster_gcp.pipes.PipesDataprocJobClient'>`class` dagster_gcp.pipes.PipesDataprocJobClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/pipes/clients/dataproc_job.py#L69' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.pipes.PipesDataprocJobClient" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A pipes client for running workloads on GCP Dataproc in Job mode.
Parameters:
- <strong>client</strong> (<em>Optional</em><em>[</em><em>google.cloud.dataproc_v1.JobControllerClient</em><em>]</em>) – The GCP Dataproc client to use.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the GCP Dataproc job. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> ([*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)) – A message reader to use to read messages from the GCP Dataproc job. For example, [`PipesGCSMessageReader`](#dagster_gcp.pipes.PipesGCSMessageReader).
- <strong>forward_termination</strong> (<em>bool</em>) – Whether to cancel the GCP Dataproc job if the Dagster process receives a termination signal.
- <strong>poll_interval</strong> (<em>float</em>) – The interval in seconds to poll the GCP Dataproc job for status updates. Defaults to 5 seconds.
<dl>
<dt><Link class="anchor" id='dagster_gcp.pipes.PipesDataprocJobClient.run'>run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/pipes/clients/dataproc_job.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.pipes.PipesDataprocJobClient.run" class="hash-link"></a></Link></dt>
<dd>
Run a job on GCP Dataproc, enriched with the pipes protocol.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The context of the currently executing Dagster op or asset.
- <strong>submit_job_params</strong> (<em>SubmitJobParams</em>) – Parameters for the `JobControllerClient.submit_job` call. See [Google Cloud SDK Documentation](https://cloud.google.com/python/docs/reference/dataproc/latest/google.cloud.dataproc_v1.services.job_controller.JobControllerClient#google_cloud_dataproc_v1_services_job_controller_JobControllerClient_submit_job)
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Additional information to pass to the Pipes session in the external process.
Returns: Wrapper containing results reported by the external
process.Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="context-injectors">
### Context injectors
<dl>
<dt><Link class="anchor" id='dagster_gcp.pipes.PipesGCSContextInjector'>`class` dagster_gcp.pipes.PipesGCSContextInjector <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/pipes/context_injectors.py#L16' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.pipes.PipesGCSContextInjector" class="hash-link"></a></Link></dt>
<dd>
A context injector that injects context by writing to a temporary GCS location.
Parameters:
- <strong>bucket</strong> (<em>str</em>) – The GCS bucket to write to.
- <strong>client</strong> (<em>google.cloud.storage.Client</em>) – A Google Cloud SDK client to use to write to GCS.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the GCS key. Will be concatenated with a random string.
</dd>
</dl>
</div>
<div class="section" id="message-readers">
### Message readers
<dl>
<dt><Link class="anchor" id='dagster_gcp.pipes.PipesGCSMessageReader'>`class` dagster_gcp.pipes.PipesGCSMessageReader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/pipes/message_readers.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.pipes.PipesGCSMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages by periodically reading message chunks from a specified GCS
bucket.
If <cite>log_readers</cite> is passed, this reader will also start the passed readers
when the first message is received from the external process.
Parameters:
- <strong>interval</strong> (<em>float</em>) – interval in seconds between attempts to download a chunk
- <strong>bucket</strong> (<em>str</em>) – The GCS bucket to read from.
- <strong>client</strong> (<em>Optional</em><em>[</em><em>cloud.google.storage.Client</em><em>]</em>) – The GCS client to use.
- <strong>log_readers</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>PipesLogReader</em><em>]</em><em>]</em>) – A set of log readers for logs on GCS.
- <strong>include_stdio_in_messages</strong> (<em>bool</em>) – Whether to send stdout/stderr to Dagster via Pipes messages. Defaults to False.
</dd>
</dl>
</div></div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_gcp.ConfigurablePickledObjectGCSIOManager'>dagster_gcp.ConfigurablePickledObjectGCSIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/io_manager.py#L169' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.ConfigurablePickledObjectGCSIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 2.0.
Please use GCSPickleIOManager instead..
:::
Renamed to GCSPickleIOManager. See GCSPickleIOManager for documentation.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.bigquery_resource'>dagster_gcp.bigquery_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/resources.py#L97' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.bigquery_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.build_bigquery_io_manager'>dagster_gcp.build_bigquery_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/bigquery/io_manager.py#L34' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.build_bigquery_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Builds an I/O manager definition that reads inputs from and writes outputs to BigQuery.
Parameters:
- <strong>type_handlers</strong> (<em>Sequence</em><em>[</em><em>DbTypeHandler</em><em>]</em>) – Each handler defines how to translate between slices of BigQuery tables and an in-memory type - e.g. a Pandas DataFrame. If only one DbTypeHandler is provided, it will be used as the default_load_type.
- <strong>default_load_type</strong> (<em>Type</em>) – When an input has no type annotation, load it as this type.
Returns: IOManagerDefinition
Examples:
```python
from dagster_gcp import build_bigquery_io_manager
from dagster_bigquery_pandas import BigQueryPandasTypeHandler
from dagster import Definitions
@asset(
key_prefix=["my_prefix"],
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
@asset(
key_prefix=["my_dataset"] # my_dataset will be used as the dataset in BigQuery
)
def my_second_table() -> pd.DataFrame: # the name of the asset will be the table name
...
bigquery_io_manager = build_bigquery_io_manager([BigQueryPandasTypeHandler()])
Definitions(
assets=[my_table, my_second_table],
resources={
"io_manager": bigquery_io_manager.configured({
"project" : {"env": "GCP_PROJECT"}
})
}
)
```
You can set a default dataset to store the assets using the `dataset` configuration value of the BigQuery I/O
Manager. This dataset will be used if no other dataset is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": bigquery_io_manager.configured({
"project" : {"env": "GCP_PROJECT"}
"dataset": "my_dataset"
})
}
)
```
On individual assets, you an also specify the dataset where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pd.DataFrame:
...
@asset(
# note that the key needs to be "schema"
metadata={"schema": "my_dataset"} # will be used as the dataset in BigQuery
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the dataset can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the dataset will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata `columns` to the
[`In`](../../../api/dagster/ops.mdx#dagster.In) or [`AssetIn`](../../../api/dagster/assets.mdx#dagster.AssetIn).
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the `gcp_credentials` configuration.
Dagster willstore this key in a temporary file and set `GOOGLE_APPLICATION_CREDENTIALS` to point to the file.
After the run completes, the file will be deleted, and `GOOGLE_APPLICATION_CREDENTIALS` will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded with this shell command: `cat $GOOGLE_APPLICATION_CREDENTIALS | base64`
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.gcs_resource'>dagster_gcp.gcs_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/resources.py#L37' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.gcs_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.gcs_pickle_io_manager'>dagster_gcp.gcs_pickle_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/io_manager.py#L179' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.gcs_pickle_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Persistent IO manager using GCS for storage.
Serializes objects via pickling. Suitable for objects storage for distributed executors, so long
as each execution node has network connectivity and credentials for GCS and the backing bucket.
Assigns each op output to a unique filepath containing run ID, step key, and output name.
Assigns each asset to a single filesystem path, at `\<base_dir>/\<asset_key>`. If the asset key
has multiple components, the final component is used as the name of the file, and the preceding
components as parent directories under the base_dir.
Subsequent materializations of an asset will overwrite previous materializations of that asset.
With a base directory of `/my/base/path`, an asset with key
`AssetKey(["one", "two", "three"])` would be stored in a file called `three` in a directory
with path `/my/base/path/one/two/`.
Example usage:
1. Attach this IO manager to a set of assets.
```python
from dagster import Definitions, asset
from dagster_gcp.gcs import gcs_pickle_io_manager, gcs_resource
@asset
def asset1():
# create df ...
return df
@asset
def asset2(asset1):
return asset1[:5]
Definitions(
assets=[asset1, asset2],
resources={
"io_manager": gcs_pickle_io_manager.configured(
{"gcs_bucket": "my-cool-bucket", "gcs_prefix": "my-cool-prefix"}
),
"gcs": gcs_resource.configured({"project": "my-cool-project"}),
},
)
```
2. Attach this IO manager to your job to make it available to your ops.
```python
from dagster import job
from dagster_gcp.gcs import gcs_pickle_io_manager, gcs_resource
@job(
resource_defs={
"io_manager": gcs_pickle_io_manager.configured(
{"gcs_bucket": "my-cool-bucket", "gcs_prefix": "my-cool-prefix"}
),
"gcs": gcs_resource.configured({"project": "my-cool-project"}),
},
)
def my_job():
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.gcs_file_manager'>dagster_gcp.gcs_file_manager ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/resources.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.gcs_file_manager" class="hash-link"></a></Link></dt>
<dd>
FileManager that provides abstract access to GCS.
Implements the [`FileManager`](../../../api/dagster/internals.mdx#dagster._core.storage.file_manager.FileManager) API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.dataproc_resource'>dagster_gcp.dataproc_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/dataproc/resources.py#L277' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.dataproc_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
</dd>
</dl>
</div>
<div class="section" id="components">
## Components
Resource components for use with Dagster’s component system.
<div class="section" id="resource-components">
### Resource components
<dl>
<dt><Link class="anchor" id='dagster_gcp.BigQueryResourceComponent'>`class` dagster_gcp.BigQueryResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/components/bigquery.py#L8' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.BigQueryResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a BigQueryResource for interacting with Google BigQuery.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSResourceComponent'>`class` dagster_gcp.GCSResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/components/gcs.py#L8' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a GCSResource for interacting with Google Cloud Storage.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.GCSFileManagerResourceComponent'>`class` dagster_gcp.GCSFileManagerResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/components/gcs.py#L36' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.GCSFileManagerResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a GCSFileManagerResource.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_gcp.DataprocResourceComponent'>`class` dagster_gcp.DataprocResourceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-gcp/dagster_gcp/components/dataproc.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_gcp.DataprocResourceComponent" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
A component that provides a DataprocResource.
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-github library'
sidebar_position: 1000
title_meta: 'dagster-github library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-github library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-github-library">
# dagster-github library
This library provides an integration with GitHub Apps, to support performing various automation
operations within your github repositories and with the tighter permissions scopes that github apps
allow for vs using a personal token.
Presently, it provides a thin wrapper on the [github v4 graphql API](https://developer.github.com/v4).
To use this integration, you’ll first need to create a GitHub App for it.
1. <strong>Create App</strong>: Follow the instructions in [https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment/](https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment), You will end up with a private key and App ID, which will be used when configuring the `dagster-github` resource. <strong>Note</strong> you will need to grant your app the relevent permissions for the API requests you want to make, for example to post issues it will need read/write access for the issues repository permission, more info on GitHub application permissions can be found [here](https://developer.github.com/v3/apps/permissions)
2. <strong>Install App</strong>: Follow the instructions in [https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment/#step-7-install-the-app-on-your-account](https://developer.github.com/apps/quickstart-guides/setting-up-your-development-environment/#step-7-install-the-app-on-your-account)
3. <strong>Find your installation_id</strong>: You can pull this from the GitHub app administration page, `https://github.com/apps/<app-name>/installations/<installation_id>`. <strong>Note</strong> if your app is installed more than once you can also programatically retrieve these IDs.
Sharing your App ID and Installation ID is fine, but make sure that the Private Key for your app is
stored securily.
<div class="section" id="posting-github-issues">
## Posting GitHub issues
You can create GitHub issues from Dagster with the GitHub resource:
```python
import os
from dagster import job, op
from dagster_github import GithubResource
@op
def github_op(github: GithubResource):
github.get_client().create_issue(
repo_name='dagster',
repo_owner='dagster-io',
title='Dagster\'s first github issue',
body='this open source thing seems like a pretty good idea',
)
@job(resource_defs={
'github': GithubResource(
github_app_id=os.getenv('GITHUB_APP_ID'),
github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
github_installation_id=os.getenv('GITHUB_INSTALLATION_ID')
)})
def github_job():
github_op()
github_job.execute_in_process()
```
Run the above code, and you’ll see the issue appear in GitHub:
GitHub enterprise users can provide their hostname in the run config. Provide `github_hostname`
as part of your github config like below.
```python
GithubResource(
github_app_id=os.getenv('GITHUB_APP_ID'),
github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
github_installation_id=os.getenv('GITHUB_INSTALLATION_ID'),
github_hostname=os.getenv('GITHUB_HOSTNAME'),
)
```
By provisioning `GithubResource` as a Dagster resource, you can post to GitHub from
within any asset or op execution.
</div>
<div class="section" id="executing-graphql-queries">
## Executing GraphQL queries
```python
import os
from dagster import job, op
from dagster_github import github_resource
@op
def github_op(github: GithubResource):
github.get_client().execute(
query="""
query get_repo_id($repo_name: String!, $repo_owner: String!) {
repository(name: $repo_name, owner: $repo_owner) {
id
}
}
""",
variables={"repo_name": repo_name, "repo_owner": repo_owner},
)
@job(resource_defs={
'github': GithubResource(
github_app_id=os.getenv('GITHUB_APP_ID'),
github_app_private_rsa_key=os.getenv('GITHUB_PRIVATE_KEY'),
github_installation_id=os.getenv('GITHUB_INSTALLATION_ID')
)})
def github_job():
github_op()
github_job.execute_in_process()
```
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient'>`class` dagster_github.resources.GithubClient <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L104' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient" class="hash-link"></a></Link></dt>
<dd>
:::warning deprecated
This API will be removed in version 0.27.
`GithubClient` is deprecated. Use your own resource and client instead. Learn how to create your own resource here: https://docs.dagster.io/guides/build/external-resources/defining-resources.
:::
A client for interacting with the GitHub API.
This client handles authentication and provides methods for making requests
to the GitHub API using an authenticated session.
Parameters:
- <strong>client</strong> (<em>requests.Session</em>) – The HTTP session used for making requests.
- <strong>app_id</strong> (<em>int</em>) – The GitHub App ID.
- <strong>app_private_rsa_key</strong> (<em>str</em>) – The private RSA key for the GitHub App.
- <strong>default_installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The default installation ID for the GitHub App.
- <strong>hostname</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The GitHub hostname, defaults to None.
- <strong>installation_tokens</strong> (<em>Dict</em><em>[</em><em>Any</em><em>, </em><em>Any</em><em>]</em>) – A dictionary to store installation tokens.
- <strong>app_token</strong> (<em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary to store the app token.
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient.create_issue'>create_issue <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L291' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient.create_issue" class="hash-link"></a></Link></dt>
<dd>
Create a new issue in the specified GitHub repository.
This method first retrieves the repository ID using the provided repository name
and owner, then creates a new issue in that repository with the given title and body.
Parameters:
- <strong>repo_name</strong> (<em>str</em>) – The name of the repository where the issue will be created.
- <strong>repo_owner</strong> (<em>str</em>) – The owner of the repository where the issue will be created.
- <strong>title</strong> (<em>str</em>) – The title of the issue.
- <strong>body</strong> (<em>str</em>) – The body content of the issue.
- <strong>installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The installation ID to use for authentication.
Returns: The response data from the GitHub API containing the created issue details.Return type: Dict[str, Any]Raises: <strong>RuntimeError</strong> – If there are errors in the response from the GitHub API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient.create_pull_request'>create_pull_request <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L383' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient.create_pull_request" class="hash-link"></a></Link></dt>
<dd>
Create a new pull request in the specified GitHub repository.
This method creates a pull request from the head reference (branch) to the base reference (branch)
in the specified repositories. It uses the provided title and body for the pull request description.
Parameters:
- <strong>base_repo_name</strong> (<em>str</em>) – The name of the base repository where the pull request will be created.
- <strong>base_repo_owner</strong> (<em>str</em>) – The owner of the base repository.
- <strong>base_ref_name</strong> (<em>str</em>) – The name of the base reference (branch) to which the changes will be merged.
- <strong>head_repo_name</strong> (<em>str</em>) – The name of the head repository from which the changes will be taken.
- <strong>head_repo_owner</strong> (<em>str</em>) – The owner of the head repository.
- <strong>head_ref_name</strong> (<em>str</em>) – The name of the head reference (branch) from which the changes will be taken.
- <strong>title</strong> (<em>str</em>) – The title of the pull request.
- <strong>body</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The body content of the pull request. Defaults to None.
- <strong>maintainer_can_modify</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether maintainers can modify the pull request. Defaults to None.
- <strong>draft</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether the pull request is a draft. Defaults to None.
- <strong>installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The installation ID to use for authentication.
Returns: The response data from the GitHub API containing the created pull request details.Return type: Dict[str, Any]Raises: <strong>RuntimeError</strong> – If there are errors in the response from the GitHub API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient.create_ref'>create_ref <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L334' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient.create_ref" class="hash-link"></a></Link></dt>
<dd>
Create a new reference (branch) in the specified GitHub repository.
This method first retrieves the repository ID and the source reference (branch or tag)
using the provided repository name, owner, and source reference. It then creates a new
reference (branch) in that repository with the given target name.
Parameters:
- <strong>repo_name</strong> (<em>str</em>) – The name of the repository where the reference will be created.
- <strong>repo_owner</strong> (<em>str</em>) – The owner of the repository where the reference will be created.
- <strong>source</strong> (<em>str</em>) – The source reference (branch or tag) from which the new reference will be created.
- <strong>target</strong> (<em>str</em>) – The name of the new reference (branch) to be created.
- <strong>installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The installation ID to use for authentication.
Returns: The response data from the GitHub API containing the created reference details.Return type: Dict[str, Any]Raises: <strong>RuntimeError</strong> – If there are errors in the response from the GitHub API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L235' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient.execute" class="hash-link"></a></Link></dt>
<dd>
Execute a GraphQL query against the GitHub API.
This method sends a POST request to the GitHub API with the provided GraphQL query
and optional variables. It ensures that the appropriate installation token is included
in the request headers.
Parameters:
- <strong>query</strong> (<em>str</em>) – The GraphQL query string to be executed.
- <strong>variables</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Optional variables to include in the query.
- <strong>headers</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Optional headers to include in the request.
- <strong>installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The installation ID to use for authentication.
Returns: The response data from the GitHub API.Return type: Dict[str, Any]Raises:
- <strong>RuntimeError</strong> – If no installation ID is provided and no default installation ID is set.
- <strong>requests.exceptions.HTTPError</strong> – If the request to the GitHub API fails.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubClient.get_installations'>get_installations <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L173' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubClient.get_installations" class="hash-link"></a></Link></dt>
<dd>
Retrieve the list of installations for the authenticated GitHub App.
This method makes a GET request to the GitHub API to fetch the installations
associated with the authenticated GitHub App. It ensures that the app token
is valid and includes it in the request headers.
Parameters: <strong>headers</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Optional headers to include in the request.Returns: A dictionary containing the installations data.Return type: Dict[str, Any]Raises: <strong>requests.exceptions.HTTPError</strong> – If the request to the GitHub API fails.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_github.resources.GithubResource'>dagster_github.resources.GithubResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L449' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.GithubResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 0.27.
`GithubResource` is deprecated. Use your own resource instead. Learn how to create your own resource here: https://docs.dagster.io/guides/build/external-resources/defining-resources.
:::
A resource configuration class for GitHub integration.
This class provides configuration fields for setting up a GitHub Application,
including the application ID, private RSA key, installation ID, and hostname.
Parameters:
- <strong>github_app_id</strong> (<em>int</em>) – The GitHub Application ID. For more information, see [https://developer.github.com/apps/](https://developer.github.com/apps/).
- <strong>github_app_private_rsa_key</strong> (<em>str</em>) – The private RSA key text for the GitHub Application. For more information, see [https://developer.github.com/apps/](https://developer.github.com/apps/).
- <strong>github_installation_id</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The GitHub Application Installation ID. Defaults to None. For more information, see [https://developer.github.com/apps/](https://developer.github.com/apps/).
- <strong>github_hostname</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The GitHub hostname. Defaults to <cite>api.github.com</cite>. For more information, see [https://developer.github.com/apps/](https://developer.github.com/apps/).
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_github.resources.github_resource'>dagster_github.resources.github_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-github/dagster_github/resources.py#L523' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_github.resources.github_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::warning deprecated
This API will be removed in version 0.27.
`github_resource` is deprecated. Use your own resource instead. Learn how to create your own resource here: https://docs.dagster.io/guides/build/external-resources/defining-resources.
:::
</dd>
</dl>
</div></div>
---
---
title: 'dagster-ge library'
sidebar_position: 1000
title_meta: 'dagster-ge library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-ge library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-ge-library">
# dagster-ge library
<dl>
<dt><Link class="anchor" id='dagster_ge.ge_validation_op_factory'>dagster_ge.ge_validation_op_factory<a href="#dagster_ge.ge_validation_op_factory" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Generates ops for interacting with [Great Expectations](https://greatexpectations.io/).
Parameters:
- - <strong>name</strong> (<em>str</em>) – the name of the op
- <strong>datasource_name</strong> (<em>str</em>) – the name of your DataSource, see your great_expectations.yml
- <strong>data_connector_name</strong> (<em>str</em>) – the name of the data connector for this datasource. This should point to a RuntimeDataConnector. For information on how to set this up, see: [https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/how_to_create_a_batch_of_data_from_an_in_memory_spark_or_pandas_dataframe](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/how_to_create_a_batch_of_data_from_an_in_memory_spark_or_pandas_dataframe)
- <strong>data_asset_name</strong> (<em>str</em>) – the name of the data asset that this op will be validating.
- <strong>suite_name</strong> (<em>str</em>) – the name of your expectation suite, see your great_expectations.yml
- <strong>batch_identifiers</strong> (<em>dict</em>) – a dictionary of batch identifiers to uniquely identify this batch of data, see: [https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/how_to_create_a_batch_of_data_from_an_in_memory_spark_or_pandas_dataframe](https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/how_to_create_a_batch_of_data_from_an_in_memory_spark_or_pandas_dataframe)
- <strong>input_dagster_type</strong> ([*DagsterType*](../../../api/dagster/types.mdx#dagster.DagsterType)) – the Dagster type used to type check the input to the op. Defaults to <cite>dagster_pandas.DataFrame</cite>.
- <strong>runtime_method_type</strong> (<em>str</em>) – how GE should interperet the op input. One of (“batch_data”, “path”, “query”). Defaults to “batch_data”, which will interperet the input as an in-memory object.
<strong>extra_kwargs</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) –
adds extra kwargs to the invocation of <cite>ge_data_context</cite>’s <cite>get_validator</cite> method. If not set, input will be:
>
```default
{ "datasource_name": datasource_name, "data_connector_name": data_connector_name, "data_asset_name": data_asset_name, "runtime_parameters": { "<runtime_method_type>": <op input> }, "batch_identifiers": batch_identifiers, "expectation_suite_name": suite_name, }
```
Returns: An op that takes in a set of data and yields both an expectation with relevant metadata and
an output with all the metadata (for user processing)
</dd>
</dl>
</div>
---
---
title: 'dagster-iceberg library'
sidebar_position: 1000
title_meta: 'dagster-iceberg library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-iceberg library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-iceberg-library">
# dagster-iceberg library
This library provides an integration with the [Iceberg](https://iceberg.apache.org) table
format.
For more information on getting started, see the [Dagster & Iceberg](https://docs.dagster.io/integrations/libraries/iceberg) documentation.
<strong>Note:</strong> This is a community-supported integration. For support, see the [Dagster Community Integrations repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-iceberg).
<div class="section" id="i-o-managers">
## I/O managers
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.arrow.PyArrowIcebergIOManager'>dagster_iceberg.io_manager.arrow.PyArrowIcebergIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/arrow.py#L56' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.arrow.PyArrowIcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
An I/O manager definition that reads inputs from and writes outputs to Iceberg tables using PyArrow.
Examples:
```python
import pandas as pd
import pyarrow as pa
from dagster import Definitions, asset
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManager
CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"
CATALOG_WAREHOUSE = (
"file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse"
)
resources = {
"io_manager": PyArrowIcebergIOManager(
name="test",
config=IcebergCatalogConfig(
properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE}
),
namespace="dagster",
)
}
@asset
def iris_dataset() -> pa.Table:
pa.Table.from_pandas(
pd.read_csv(
"https://docs.dagster.io/assets/iris.csv",
names=[
"sepal_length_cm",
"sepal_width_cm",
"petal_length_cm",
"petal_width_cm",
"species",
],
)
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”. The I/O manager will check if the namespace
exists in the Iceberg catalog. It does not automatically create the namespace if it does not exist.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pa.Table:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
`In` or `AssetIn`.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pa.Table):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.daft.DaftIcebergIOManager'>dagster_iceberg.io_manager.daft.DaftIcebergIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/daft.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.daft.DaftIcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
An I/O manager definition that reads inputs from and writes outputs to Iceberg tables using Daft.
Examples:
```python
import daft as da
import pandas as pd
from dagster import Definitions, asset
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.daft import DaftIcebergIOManager
CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"
CATALOG_WAREHOUSE = (
"file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse"
)
resources = {
"io_manager": DaftIcebergIOManager(
name="test",
config=IcebergCatalogConfig(
properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE}
),
namespace="dagster",
)
}
@asset
def iris_dataset() -> da.DataFrame:
return da.from_pandas(
pd.read_csv(
"https://docs.dagster.io/assets/iris.csv",
names=[
"sepal_length_cm",
"sepal_width_cm",
"petal_length_cm",
"petal_width_cm",
"species",
],
)
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”. The I/O manager will check if the namespace
exists in the Iceberg catalog. It does not automatically create the namespace if it does not exist.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> da.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
`In` or `AssetIn`.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: da.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.pandas.PandasIcebergIOManager'>dagster_iceberg.io_manager.pandas.PandasIcebergIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/pandas.py#L43' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.pandas.PandasIcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
An I/O manager definition that reads inputs from and writes outputs to Iceberg tables using pandas.
Examples:
```python
import pandas as pd
from dagster import Definitions, asset
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.pandas import PandasIcebergIOManager
CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"
CATALOG_WAREHOUSE = (
"file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse"
)
resources = {
"io_manager": PandasIcebergIOManager(
name="test",
config=IcebergCatalogConfig(
properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE}
),
namespace="dagster",
)
}
@asset
def iris_dataset() -> pd.DataFrame:
return pd.read_csv(
"https://docs.dagster.io/assets/iris.csv",
names=[
"sepal_length_cm",
"sepal_width_cm",
"petal_length_cm",
"petal_width_cm",
"species",
],
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”. The I/O manager will check if the namespace
exists in the Iceberg catalog. It does not automatically create the namespace if it does not exist.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
`In` or `AssetIn`.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.polars.PolarsIcebergIOManager'>dagster_iceberg.io_manager.polars.PolarsIcebergIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/polars.py#L58' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.polars.PolarsIcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
An I/O manager definition that reads inputs from and writes outputs to Iceberg tables using Polars.
Examples:
```python
import polars as pl
from dagster import Definitions, asset
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.polars import PolarsIcebergIOManager
CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"
CATALOG_WAREHOUSE = (
"file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse"
)
resources = {
"io_manager": PolarsIcebergIOManager(
name="test",
config=IcebergCatalogConfig(
properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE}
),
namespace="dagster",
)
}
@asset
def iris_dataset() -> pl.DataFrame:
return pl.read_csv(
"https://docs.dagster.io/assets/iris.csv",
has_header=False,
new_columns=[
"sepal_length_cm",
"sepal_width_cm",
"petal_length_cm",
"petal_width_cm",
"species",
],
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”. The I/O manager will check if the namespace
exists in the Iceberg catalog. It does not automatically create the namespace if it does not exist.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pl.DataFrame:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
`In` or `AssetIn`.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.spark.SparkIcebergIOManager'>dagster_iceberg.io_manager.spark.SparkIcebergIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/spark.py#L145' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.spark.SparkIcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
An I/O manager definition that reads inputs from and writes outputs to Iceberg tables using PySpark.
This I/O manager is only designed to work with Spark Connect.
Example:
```python
from dagster import Definitions, asset
from dagster_iceberg.io_manager.spark import SparkIcebergIOManager
from pyspark.sql import SparkSession
from pyspark.sql.connect.dataframe import DataFrame
resources = {
"io_manager": SparkIcebergIOManager(
catalog_name="test",
namespace="dagster",
remote_url="spark://localhost",
)
}
@asset
def iris_dataset() -> DataFrame:
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
return spark.read.csv(
"https://docs.dagster.io/assets/iris.csv",
schema=(
"sepal_length_cm FLOAT, "
"sepal_width_cm FLOAT, "
"petal_length_cm FLOAT, "
"petal_width_cm FLOAT, "
"species STRING"
),
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
</dd>
</dl>
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_iceberg.resource.IcebergTableResource'>dagster_iceberg.resource.IcebergTableResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/resource.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.resource.IcebergTableResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Resource for interacting with a PyIceberg table.
Example:
```python
from dagster import Definitions, asset
from dagster_iceberg import IcebergTableResource
@asset
def my_table(iceberg_table: IcebergTableResource):
df = iceberg_table.load().to_pandas()
warehouse_path = "/path/to/warehouse"
defs = Definitions(
assets=[my_table],
resources={
"iceberg_table": IcebergTableResource(
name="my_catalog",
config=IcebergCatalogConfig(
properties={
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
}
),
table="my_table",
namespace="my_namespace",
)
},
)
```
</dd>
</dl>
</div>
<div class="section" id="config">
## Config
<dl>
<dt><Link class="anchor" id='dagster_iceberg.config.IcebergCatalogConfig'>`class` dagster_iceberg.config.IcebergCatalogConfig <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/config.py#L14' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.config.IcebergCatalogConfig" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Configuration for Iceberg Catalogs.
See the [Catalogs section](https://py.iceberg.apache.org/configuration/#catalogs)
for configuration options.
You can configure the Iceberg IO manager:
>
1. Using a `.pyiceberg.yaml` configuration file.
2. Through environment variables.
3. Using the `IcebergCatalogConfig` configuration object.
For more information about the first two configuration options, see
[Setting Configuration Values](https://py.iceberg.apache.org/configuration/#setting-configuration-values).
Example:
```python
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManager
warehouse_path = "/path/to/warehouse"
io_manager = PyArrowIcebergIOManager(
name="my_catalog",
config=IcebergCatalogConfig(
properties={
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
}
),
namespace="my_namespace",
)
```
</dd>
</dl>
</div>
<div class="section" id="base-classes">
## Base classes
<dl>
<dt><Link class="anchor" id='dagster_iceberg.io_manager.base.IcebergIOManager'>`class` dagster_iceberg.io_manager.base.IcebergIOManager <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/io_manager/base.py#L105' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.io_manager.base.IcebergIOManager" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Base class for an I/O manager definition that reads inputs from and writes outputs to Iceberg tables.
Examples:
```python
import pandas as pd
import pyarrow as pa
from dagster import Definitions, asset
from dagster_iceberg.config import IcebergCatalogConfig
from dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManager
CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"
CATALOG_WAREHOUSE = (
"file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse"
)
resources = {
"io_manager": PyArrowIcebergIOManager(
name="test",
config=IcebergCatalogConfig(
properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE}
),
namespace="dagster",
)
}
@asset
def iris_dataset() -> pa.Table:
pa.Table.from_pandas(
pd.read_csv(
"https://docs.dagster.io/assets/iris.csv",
names=[
"sepal_length_cm",
"sepal_width_cm",
"petal_length_cm",
"petal_width_cm",
"species",
],
)
)
defs = Definitions(assets=[iris_dataset], resources=resources)
```
If you do not provide a schema, Dagster will determine a schema based on the assets and ops using
the I/O manager. For assets, the schema will be determined from the asset key, as in the above example.
For ops, the schema can be specified by including a “schema” entry in output metadata. If none
of these is provided, the schema will default to “public”. The I/O manager will check if the namespace
exists in the Iceberg catalog. It does not automatically create the namespace if it does not exist.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pa.Table:
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
`In` or `AssetIn`.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pa.Table):
# my_table will just contain the data from column "a"
...
```
To select a write mode, set the `write_mode` key in the asset definition metadata or at runtime via output metadata.
Write mode set at runtime takes precedence over the one set in the definition metadata.
Valid modes are `append`, `overwrite`, and `upsert`; default is `overwrite`.
```python
# set at definition time via definition metadata
@asset(
metadata={"write_mode": "append"}
)
def my_table_a(my_table: pa.Table):
return my_table
# set at runtime via output metadata
@asset
def my_table_a(context: AssetExecutionContext, my_table: pa.Table):
# my_table will be written with append mode
context.add_output_metadata({"write_mode": "append"})
return my_table
```
To use upsert mode, set `write_mode` to `upsert` and provide `upsert_options` in asset definition metadata
or output metadata. The `upsert_options` dictionary should contain `join_cols` (list of columns to join on),
`when_matched_update_all` (boolean), and `when_not_matched_insert_all` (boolean).
Upsert options set at runtime take precedence over those set in definition metadata.
```python
# set at definition time via definition metadata
@asset(
metadata={
"write_mode": "upsert",
"upsert_options": {
"join_cols": ["id"],
"when_matched_update_all": True,
"when_not_matched_insert_all": True,
}
}
)
def my_table_upsert(my_table: pa.Table):
return my_table
# set at runtime via output metadata (overrides definition metadata)
@asset(
metadata={
"write_mode": "upsert",
"upsert_options": {
"join_cols": ["id"],
"when_matched_update_all": True,
"when_not_matched_insert_all": False,
}
}
)
def my_table_upsert_dynamic(context: AssetExecutionContext, my_table: pa.Table):
# Override upsert options at runtime
context.add_output_metadata({
"upsert_options": {
"join_cols": ["id", "timestamp"],
"when_matched_update_all": False,
"when_not_matched_insert_all": False,
}
})
return my_table
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_iceberg.handler.IcebergBaseTypeHandler'>`class` dagster_iceberg.handler.IcebergBaseTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_iceberg/handler.py#L35' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_iceberg.handler.IcebergBaseTypeHandler" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Base class for a type handler that reads inputs from and writes outputs to Iceberg tables.
</dd>
</dl>
</div></div>
---
---
title: 'dagstermill library'
sidebar_position: 1000
title_meta: 'dagstermill library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagstermill library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagstermill-library">
# dagstermill library
This library provides an integration with <cite>papermill</cite> to allow you to run Jupyter notebooks with Dagster.
Related guides:
- [Using Jupyter notebooks with Papermill and Dagster](https://docs.dagster.io/integrations/libraries/jupyter)
<dl>
<dt><Link class="anchor" id='dagstermill.define_dagstermill_asset'>dagstermill.define_dagstermill_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/asset_factory.py#L73' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.define_dagstermill_asset" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Creates a Dagster asset for a Jupyter notebook.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name for the asset
- <strong>notebook_path</strong> (<em>str</em>) – Path to the backing notebook
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – If provided, the asset’s key is the concatenation of the key_prefix and the asset’s name, which defaults to the name of the decorated function. Each item in key_prefix must be a valid name in dagster (ie only contains letters, numbers, and _) and may not contain python reserved keywords.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*AssetIn*](../../../api/dagster/assets.mdx#dagster.AssetIn)<em>]</em><em>]</em>) – A dictionary that maps input names to information about the input.
- <strong>deps</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Union</em><em>[</em>[*AssetsDefinition*](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)<em>, </em>[*SourceAsset*](../../../api/dagster/assets.mdx#dagster.SourceAsset)<em>, </em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – The assets that are upstream dependencies, but do not pass an input value to the notebook.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](../../../api/dagster/config.mdx#dagster.ConfigSchema)) – The configuration schema for the asset’s underlying op. If set, Dagster will check that config provided for the op matches this schema and fail if it does not. If not set, Dagster will accept any config provided for the op.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dict of metadata entries for the asset.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Set of resource handles required by the notebook.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Description of the asset to display in the Dagster UI.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – Defines the set of partition keys that compose the asset.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags for the op that computes the asset. Frameworks may expect and require certain metadata to be attached to a op. Values that are not strings will be json encoded and must meet the criteria that <cite>json.loads(json.dumps(value)) == value</cite>.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string name used to organize multiple assets into groups. If not provided, the name “default” is used.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](../../../api/dagster/resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – <span className="flag flag-info">beta</span> (Beta) A mapping of resource keys to resource definitions. These resources will be initialized during execution, and can be accessed from the context within the notebook.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string key for the IO manager used to store the output notebook. If not provided, the default key output_notebook_io_manager will be used.
- <strong>retry_policy</strong> (<em>Optional</em><em>[</em>[*RetryPolicy*](../../../api/dagster/ops.mdx#dagster.RetryPolicy)<em>]</em>) – The retry policy for the op that computes the asset.
- <strong>save_notebook_on_failure</strong> (<em>bool</em>) – If True and the notebook fails during execution, the failed notebook will be written to the Dagster storage directory. The location of the file will be printed in the Dagster logs. Defaults to False.
- <strong>asset_tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A dictionary of tags to apply to the asset.
- <strong>non_argument_deps</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Set</em><em>[</em>[*AssetKey*](../../../api/dagster/assets.mdx#dagster.AssetKey)<em>]</em><em>, </em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – Deprecated, use deps instead. Set of asset keys that are upstream dependencies, but do not pass an input to the asset.
Examples:
```python
from dagstermill import define_dagstermill_asset
from dagster import asset, AssetIn, AssetKey
from sklearn import datasets
import pandas as pd
import numpy as np
@asset
def iris_dataset():
sk_iris = datasets.load_iris()
return pd.DataFrame(
data=np.c_[sk_iris["data"], sk_iris["target"]],
columns=sk_iris["feature_names"] + ["target"],
)
iris_kmeans_notebook = define_dagstermill_asset(
name="iris_kmeans_notebook",
notebook_path="/path/to/iris_kmeans.ipynb",
ins={
"iris": AssetIn(key=AssetKey("iris_dataset"))
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.define_dagstermill_op'>dagstermill.define_dagstermill_op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/factory.py#L380' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.define_dagstermill_op" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Wrap a Jupyter notebook in a op.
Parameters:
- <strong>name</strong> (<em>str</em>) – The name of the op.
- <strong>notebook_path</strong> (<em>str</em>) – Path to the backing notebook.
- <strong>ins</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*In*](../../../api/dagster/ops.mdx#dagster.In)<em>]</em><em>]</em>) – The op’s inputs.
- <strong>outs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*Out*](../../../api/dagster/ops.mdx#dagster.Out)<em>]</em><em>]</em>) – The op’s outputs. Your notebook should call [`yield_result()`](#dagstermill.yield_result) to yield each of these outputs.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The string names of any required resources.
- <strong>output_notebook_name</strong> – (Optional[str]): If set, will be used as the name of an injected output of type of `BufferedIOBase` that is the file object of the executed notebook (in addition to the [`AssetMaterialization`](../../../api/dagster/ops.mdx#dagster.AssetMaterialization) that is always created). It allows the downstream ops to access the executed notebook via a file object.
- <strong>asset_key_prefix</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>, </em><em>str</em><em>]</em><em>]</em>) – If set, will be used to prefix the asset keys for materialized notebooks.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – If set, description used for op.
- <strong>tags</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – If set, additional tags used to annotate op. Dagster uses the tag keys <cite>notebook_path</cite> and <cite>kind</cite>, which cannot be overwritten by the user.
- <strong>io_manager_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – If using output_notebook_name, you can additionally provide a string key for the IO manager used to store the output notebook. If not provided, the default key output_notebook_io_manager will be used.
- <strong>save_notebook_on_failure</strong> (<em>bool</em>) – If True and the notebook fails during execution, the failed notebook will be written to the Dagster storage directory. The location of the file will be printed in the Dagster logs. Defaults to False.
Returns: [`OpDefinition`](../../../api/dagster/ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.ConfigurableLocalOutputNotebookIOManager'>`class` dagstermill.ConfigurableLocalOutputNotebookIOManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/io_managers.py#L87' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.ConfigurableLocalOutputNotebookIOManager" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Built-in IO Manager for handling output notebook.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.get_context'>dagstermill.get_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/manager.py#L205' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.get_context" class="hash-link"></a></Link></dt>
<dd>
Get a dagstermill execution context for interactive exploration and development.
Parameters:
- <strong>op_config</strong> (<em>Optional</em><em>[</em><em>Any</em><em>]</em>) – If specified, this value will be made available on the context as its `op_config` property.
- <strong>resource_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*ResourceDefinition*](../../../api/dagster/resources.mdx#dagster.ResourceDefinition)<em>]</em><em>]</em>) – Specifies resources to provide to context.
- <strong>logger_defs</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em>[*LoggerDefinition*](../../../api/dagster/loggers.mdx#dagster.LoggerDefinition)<em>]</em><em>]</em>) – Specifies loggers to provide to context.
- <strong>run_config</strong> (<em>Optional</em><em>[</em><em>dict</em><em>]</em>) – The config dict with which to construct the context.
Returns: [`DagstermillExecutionContext`](#dagstermill.DagstermillExecutionContext)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.yield_event'>dagstermill.yield_event <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/manager.py#L339' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.yield_event" class="hash-link"></a></Link></dt>
<dd>
Yield a dagster event directly from notebook code.
When called interactively or in development, returns its input.
Parameters: <strong>dagster_event</strong> (Union[[`dagster.AssetMaterialization`](../../../api/dagster/ops.mdx#dagster.AssetMaterialization), [`dagster.ExpectationResult`](../../../api/dagster/ops.mdx#dagster.ExpectationResult), [`dagster.TypeCheck`](../../../api/dagster/ops.mdx#dagster.TypeCheck), [`dagster.Failure`](../../../api/dagster/ops.mdx#dagster.Failure), [`dagster.RetryRequested`](../../../api/dagster/ops.mdx#dagster.RetryRequested)]) – An event to yield back to Dagster.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.yield_result'>dagstermill.yield_result <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/manager.py#L296' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.yield_result" class="hash-link"></a></Link></dt>
<dd>
Yield a result directly from notebook code.
When called interactively or in development, returns its input.
Parameters:
- <strong>value</strong> (<em>Any</em>) – The value to yield.
- <strong>output_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the result to yield (default: `'result'`).
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext'>`class` dagstermill.DagstermillExecutionContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L21' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Dagstermill-specific execution context.
Do not initialize directly: use [`dagstermill.get_context()`](#dagstermill.get_context).
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.job_def'>`property` job_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L99' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.job_def" class="hash-link"></a></Link></dt>
<dd>
The job definition for the context.
This will be a dagstermill-specific shim.
Type: [`dagster.JobDefinition`](../../../api/dagster/jobs.mdx#dagster.JobDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L93' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.job_name" class="hash-link"></a></Link></dt>
<dd>
The name of the executing job.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.logging_tags'>`property` logging_tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L87' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.logging_tags" class="hash-link"></a></Link></dt>
<dd>
The logging tags for the context.
Type: dict
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.op_config'>`property` op_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L155' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.op_config" class="hash-link"></a></Link></dt>
<dd>
A dynamically-created type whose properties allow access to
op-specific config.
Type: collections.namedtuple
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.op_def'>`property` op_def <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L136' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.op_def" class="hash-link"></a></Link></dt>
<dd>
The op definition for the context.
In interactive contexts, this may be a dagstermill-specific shim, depending whether an
op definition was passed to `dagstermill.get_context`.
Type: [`dagster.OpDefinition`](../../../api/dagster/ops.mdx#dagster.OpDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.run'>`property` run <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L122' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.run" class="hash-link"></a></Link></dt>
<dd>
The job run for the context.
Type: [`dagster.DagsterRun`](../../../api/dagster/internals.mdx#dagster.DagsterRun)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.run_config'>`property` run_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L76' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.run_config" class="hash-link"></a></Link></dt>
<dd>
The run_config for the context.
Type: dict
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillExecutionContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/context.py#L70' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillExecutionContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The run_id for the context.
Type: str
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagstermill.DagstermillError'>`class` dagstermill.DagstermillError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagstermill/dagstermill/errors.py#L4' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagstermill.DagstermillError" class="hash-link"></a></Link></dt>
<dd>
Base class for errors raised by dagstermill.
</dd>
</dl>
</div>
---
---
title: 'dagster-k8s'
sidebar_position: 1000
title_meta: 'dagster-k8s API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-k8s Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-k8s">
# dagster-k8s
See also the [Kubernetes deployment guide](https://docs.dagster.io/deployment/oss/deployment-options/kubernetes).
This library contains utilities for running Dagster with Kubernetes. This includes a Python API
allowing the webserver to launch runs as Kubernetes Jobs, as well as a Helm chart you can use as the basis
for a Dagster deployment on a Kubernetes cluster.
<div class="section" id="apis">
## APIs
<dl>
<dt><Link class="anchor" id='dagster_k8s.K8sRunLauncher'>dagster_k8s.K8sRunLauncher RunLauncher<a href="#dagster_k8s.K8sRunLauncher" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
RunLauncher that starts a Kubernetes Job for each Dagster job run.
Encapsulates each run in a separate, isolated invocation of `dagster-graphql`.
You can configure a Dagster instance to use this RunLauncher by adding a section to your
`dagster.yaml` like the following:
```yaml
run_launcher:
module: dagster_k8s.launcher
class: K8sRunLauncher
config:
service_account_name: your_service_account
job_image: my_project/dagster_image:latest
instance_config_map: dagster-instance
postgres_password_secret: dagster-postgresql-secret
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_k8s.k8s_job_executor'>dagster_k8s.k8s_job_executor ExecutorDefinition<a href="#dagster_k8s.k8s_job_executor" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Executor which launches steps as Kubernetes Jobs.
To use the <cite>k8s_job_executor</cite>, set it as the <cite>executor_def</cite> when defining a job:
```python
from dagster_k8s import k8s_job_executor
from dagster import job
@job(executor_def=k8s_job_executor)
def k8s_job():
pass
```
Then you can configure the executor with run config as follows:
```YAML
execution:
config:
job_namespace: 'some-namespace'
image_pull_policy: ...
image_pull_secrets: ...
service_account_name: ...
env_config_maps: ...
env_secrets: ...
env_vars: ...
job_image: ... # leave out if using userDeployments
max_concurrent: ...
```
<cite>max_concurrent</cite> limits the number of pods that will execute concurrently for one run. By default
there is no limit- it will maximally parallel as allowed by the DAG. Note that this is not a
global limit.
Configuration set on the Kubernetes Jobs and Pods created by the <cite>K8sRunLauncher</cite> will also be
set on Kubernetes Jobs and Pods created by the <cite>k8s_job_executor</cite>.
Configuration set using <cite>tags</cite> on a <cite>@job</cite> will only apply to the <cite>run</cite> level. For configuration
to apply at each <cite>step</cite> it must be set using <cite>tags</cite> for each <cite>@op</cite>.
</dd>
</dl>
</div>
<div class="section" id="ops">
## Ops
<dl>
<dt><Link class="anchor" id='dagster_k8s.k8s_job_op'>dagster_k8s.k8s_job_op `=` \<dagster._core.definitions.op_definition.OpDefinition object><a href="#dagster_k8s.k8s_job_op" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
An op that runs a Kubernetes job using the k8s API.
Contrast with the <cite>k8s_job_executor</cite>, which runs each Dagster op in a Dagster job in its
own k8s job.
This op may be useful when:
- You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)
- You want to run the rest of a Dagster job using a specific executor, and only a single op in k8s.
For example:
```python
from dagster_k8s import k8s_job_op
from dagster import job
first_op = k8s_job_op.configured(
{
"image": "busybox",
"command": ["/bin/sh", "-c"],
"args": ["echo HELLO"],
},
name="first_op",
)
second_op = k8s_job_op.configured(
{
"image": "busybox",
"command": ["/bin/sh", "-c"],
"args": ["echo GOODBYE"],
},
name="second_op",
)
@job
def full_job():
second_op(first_op())
```
You can create your own op with the same implementation by calling the <cite>execute_k8s_job</cite> function
inside your own op.
The service account that is used to run this job should have the following RBAC permissions:
```YAML
rules:
- apiGroups: ["batch"]
resources: ["jobs", "jobs/status"]
verbs: ["*"]
# The empty arg "" corresponds to the core API group
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["*"]'
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_k8s.execute_k8s_job'>dagster_k8s.execute_k8s_job<a href="#dagster_k8s.execute_k8s_job" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `PipesK8sClient` instead for launching k8s pods within Dagster assets and ops..
:::
This function is a utility for executing a Kubernetes job from within a Dagster op.
Parameters:
- <strong>image</strong> (<em>str</em>) – The image in which to launch the k8s job.
- <strong>command</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The command to run in the container within the launched k8s job. Default: None.
- <strong>args</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The args for the command for the container. Default: None.
- <strong>namespace</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Override the kubernetes namespace in which to run the k8s job. Default: None.
- <strong>image_pull_policy</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Allows the image pull policy to be overridden, e.g. to facilitate local testing with [kind](https://kind.sigs.k8s.io/). Default: `"Always"`. See: [https://kubernetes.io/docs/concepts/containers/images/#updating-images](https://kubernetes.io/docs/concepts/containers/images/#updating-images).
- <strong>image_pull_secrets</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em><em>]</em>) – Optionally, a list of dicts, each of which corresponds to a Kubernetes `LocalObjectReference` (e.g., `\{'name': 'myRegistryName'}`). This allows you to specify the ``imagePullSecrets` on a pod basis. Typically, these will be provided through the service account, when needed, and you will not need to pass this argument. See: [https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) and [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#podspec-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#podspec-v1-core)
- <strong>service_account_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The name of the Kubernetes service account under which to run the Job. Defaults to “default” env_config_maps (Optional[List[str]]): A list of custom ConfigMapEnvSource names from which to draw environment variables (using `envFrom`) for the Job. Default: `[]`. See: [https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#define-an-environment-variable-for-a-container](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#define-an-environment-variable-for-a-container)
- <strong>env_secrets</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of custom Secret names from which to draw environment variables (using `envFrom`) for the Job. Default: `[]`. See: [https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)
- <strong>env_vars</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – A list of environment variables to inject into the Job. Default: `[]`. See: [https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables)
- <strong>volume_mounts</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em>[*Permissive*](../../../api/dagster/config.mdx#dagster.Permissive)<em>]</em><em>]</em>) – A list of volume mounts to include in the job’s container. Default: `[]`. See: [https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumemount-v1-core](https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumemount-v1-core)
- <strong>volumes</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em>[*Permissive*](../../../api/dagster/config.mdx#dagster.Permissive)<em>]</em><em>]</em>) – A list of volumes to include in the Job’s Pod. Default: `[]`. See: [https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volume-v1-core](https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volume-v1-core)
- <strong>labels</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – Additional labels that should be included in the Job’s Pod. See: [https://kubernetes.io/docs/concepts/overview/working-with-objects/labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels)
- <strong>resources</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – [https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
- <strong>scheduler_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Use a custom Kubernetes scheduler for launched Pods. See: [https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/)
- <strong>load_incluster_config</strong> (<em>bool</em>) – Whether the op is running within a k8s cluster. If `True`, we assume the launcher is running within the target cluster and load config using `kubernetes.config.load_incluster_config`. Otherwise, we will use the k8s config specified in `kubeconfig_file` (using `kubernetes.config.load_kube_config`) or fall back to the default kubeconfig. Default: True,
- <strong>kubeconfig_file</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The kubeconfig file from which to load config. Defaults to using the default kubeconfig. Default: None.
- <strong>timeout</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – Raise an exception if the op takes longer than this timeout in seconds to execute. Default: None.
- <strong>container_config</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s pod’s main container ([https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core)). Keys can either snake_case or camelCase.Default: None.
- <strong>pod_template_spec_metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s pod’s metadata ([https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta)). Keys can either snake_case or camelCase. Default: None.
- <strong>pod_spec_config</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s pod’s pod spec ([https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec)). Keys can either snake_case or camelCase. Default: None.
- <strong>job_metadata</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s job’s metadata ([https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta)). Keys can either snake_case or camelCase. Default: None.
- <strong>job_spec_config</strong> (<em>Optional</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s job’s job spec ([https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#jobspec-v1-batch](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#jobspec-v1-batch)). Keys can either snake_case or camelCase.Default: None.
- <strong>k8s_job_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Overrides the name of the k8s job. If not set, will be set to a unique name based on the current run ID and the name of the calling op. If set, make sure that the passed in name is a valid Kubernetes job name that does not already exist in the cluster.
- <strong>merge_behavior</strong> (<em>Optional</em><em>[</em><em>K8sConfigMergeBehavior</em><em>]</em>) – How raw k8s config set on this op should be merged with any raw k8s config set on the code location that launched the op. By default, the value is K8sConfigMergeBehavior.DEEP, meaning that the two dictionaries are recursively merged, appending list fields together and merging dictionary fields. Setting it to SHALLOW will make the dictionaries shallowly merged - any shared values in the dictionaries will be replaced by the values set on this op.
- <strong>delete_failed_k8s_jobs</strong> (<em>bool</em>) – Whether to immediately delete failed Kubernetes jobs. If False, failed jobs will remain accessible through the Kubernetes API until deleted by a user or cleaned up by the .spec.ttlSecondsAfterFinished parameter of the job. ([https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/](https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/)). Defaults to True.
</dd>
</dl>
</div>
<div class="section" id="python-api">
## Python API
The `K8sRunLauncher` allows webserver instances to be configured to launch new runs by starting
per-run Kubernetes Jobs. To configure the `K8sRunLauncher`, your `dagster.yaml` should
include a section like:
```yaml
run_launcher:
module: dagster_k8s.launcher
class: K8sRunLauncher
config:
image_pull_secrets:
service_account_name: dagster
job_image: "my-company.com/image:latest"
dagster_home: "/opt/dagster/dagster_home"
postgres_password_secret: "dagster-postgresql-secret"
image_pull_policy: "IfNotPresent"
job_namespace: "dagster"
instance_config_map: "dagster-instance"
env_config_maps:
- "dagster-k8s-job-runner-env"
env_secrets:
- "dagster-k8s-some-secret"
env_vars:
- "ENV_VAR=1"
labels:
resources:
run_k8s_config:
pod_template_spec_metadata:
pod_spec_config:
job_metadata:
job_spec_config:
container_config:
volume_mounts:
volumes:
security_context:
scheduler_name:
kubeconfig_file:
```
</div>
<div class="section" id="helm-chart">
## Helm chart
For local dev (e.g., on kind or minikube):
```shell
helm install \
--set dagsterWebserver.image.repository="dagster.io/buildkite-test-image" \
--set dagsterWebserver.image.tag="py310-latest" \
--set job_runner.image.repository="dagster.io/buildkite-test-image" \
--set job_runner.image.tag="py310-latest" \
--set imagePullPolicy="IfNotPresent" \
dagster \
helm/dagster/
```
Upon installation, the Helm chart will provide instructions for port forwarding
the Dagster webserver and Flower (if configured).
</div>
<div class="section" id="running-tests">
## Running tests
To run the unit tests:
```default
pytest -m "not integration"
```
To run the integration tests, you must have [Docker](https://docs.docker.com/install),
[kind](https://kind.sigs.k8s.io/docs/user/quick-start#installation),
and [helm](https://helm.sh/docs/intro/install) installed.
On macOS:
```default
brew install kind
brew install helm
```
Docker must be running.
You may experience slow first test runs thanks to image pulls (run `pytest -svv --fulltrace` for
visibility). Building images and loading them to the kind cluster is slow, and there is
no visibility into the progress of the load.
<strong>NOTE:</strong> This process is quite slow, as it requires bootstrapping a local `kind` cluster with
Docker images and the `dagster-k8s` Helm chart. For faster development, you can either:
1. Keep a warm kind cluster
2. Use a remote K8s cluster, e.g. via AWS EKS or GCP GKE
Instructions are below.
<div class="section" id="faster-local-development-with-kind">
### Faster local development (with kind)
You may find that the kind cluster creation, image loading, and kind cluster creation loop
is too slow for effective local dev.
You may bypass cluster creation and image loading in the following way. First add the `--no-cleanup`
flag to your pytest invocation:
```shell
pytest --no-cleanup -s -vvv -m "not integration"
```
The tests will run as before, but the kind cluster will be left running after the tests are completed.
For subsequent test runs, you can run:
```shell
pytest --kind-cluster="cluster-d9971c84d44d47f382a2928c8c161faa" --existing-helm-namespace="dagster-test-95590a" -s -vvv -m "not integration"
```
This will bypass cluster creation, image loading, and Helm chart installation, for much faster tests.
The kind cluster name and Helm namespace for this command can be found in the logs, or retrieved
via the respective CLIs, using `kind get clusters` and `kubectl get namespaces`. Note that
for `kubectl` and `helm` to work correctly with a kind cluster, you should override your
kubeconfig file location with:
```shell
kind get kubeconfig --name kind-test > /tmp/kubeconfig
export KUBECONFIG=/tmp/kubeconfig
```
<div class="section" id="manual-kind-cluster-setup">
#### Manual kind cluster setup
The test fixtures provided by `dagster-k8s` automate the process described below, but sometimes
it’s useful to manually configure a kind cluster and load images onto it.
First, ensure you have a Docker image appropriate for your Python version. Run, from the root of
the repo:
```shell
./python_modules/dagster-test/dagster_test/test_project/build.sh 3.7.6 \
dagster.io.priv/buildkite-test-image:py310-latest
```
In the above invocation, the Python majmin version should be appropriate for your desired tests.
Then run the following commands to create the cluster and load the image. Note that there is no
feedback from the loading process.
```shell
kind create cluster --name kind-test
kind load docker-image --name kind-test dagster.io/dagster-docker-buildkite:py310-latest
```
If you are deploying the Helm chart with an in-cluster Postgres (rather than an external database),
and/or with dagster-celery workers (and a RabbitMQ), you’ll also want to have images present for
rabbitmq and postgresql:
```shell
docker pull docker.io/bitnami/rabbitmq
docker pull docker.io/bitnami/postgresql
kind load docker-image --name kind-test docker.io/bitnami/rabbitmq:latest
kind load docker-image --name kind-test docker.io/bitnami/postgresql:latest
```
Then you can run pytest as follows:
```shell
pytest --kind-cluster=kind-test
```
</div></div>
<div class="section" id="faster-local-development-with-an-existing-k8s-cluster">
### Faster local development (with an existing K8s cluster)
If you already have a development K8s cluster available, you can run tests on that cluster vs.
running locally in `kind`.
For this to work, first build and deploy the test image to a registry available to your cluster.
For example, with a private ECR repository:
```default
./python_modules/dagster-test/dagster_test/test_project/build.sh 3.7.6
docker tag dagster-docker-buildkite:latest $AWS_ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/dagster-k8s-tests:2020-04-21T21-04-06
aws ecr get-login --no-include-email --region us-west-1 | sh
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-west-1.amazonaws.com/dagster-k8s-tests:2020-04-21T21-04-06
```
Then, you can run tests on EKS with:
```default
export DAGSTER_DOCKER_IMAGE_TAG="2020-04-21T21-04-06"
export DAGSTER_DOCKER_REPOSITORY="$AWS_ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com"
export DAGSTER_DOCKER_IMAGE="dagster-k8s-tests"
# First run with --no-cleanup to leave Helm chart in place
pytest --cluster-provider="kubeconfig" --no-cleanup -s -vvv
# Subsequent runs against existing Helm chart
pytest --cluster-provider="kubeconfig" --existing-helm-namespace="dagster-test-<some id>" -s -vvv
```
</div>
<div class="section" id="validating-helm-charts">
### Validating Helm charts
To test / validate Helm charts, you can run:
```shell
helm install dagster --dry-run --debug helm/dagster
helm lint
```
</div>
<div class="section" id="enabling-gcr-access-from-minikube">
### Enabling GCR access from Minikube
To enable GCR access from Minikube:
```shell
kubectl create secret docker-registry element-dev-key \
--docker-server=https://gcr.io \
--docker-username=oauth2accesstoken \
--docker-password="$(gcloud auth print-access-token)" \
--docker-email=my@email.com
```
</div>
<div class="section" id="a-note-about-pvcs">
### A note about PVCs
Both the Postgres and the RabbitMQ Helm charts will store credentials using Persistent Volume
Claims, which will outlive test invocations and calls to `helm uninstall`. These must be deleted if
you want to change credentials. To view your pvcs, run:
```default
kubectl get pvc
```
</div>
<div class="section" id="testing-redis">
### Testing Redis
The Redis Helm chart installs w/ a randomly-generated password by default; turn this off:
```default
helm install dagredis stable/redis --set usePassword=false
```
Then, to connect to your database from outside the cluster execute the following commands:
```default
kubectl port-forward --namespace default svc/dagredis-master 6379:6379
redis-cli -h 127.0.0.1 -p 6379
```
</div></div>
<div class="section" id="pipes">
## Pipes
<dl>
<dt><Link class="anchor" id='dagster_k8s.PipesK8sClient'>`class` dagster_k8s.PipesK8sClient<a href="#dagster_k8s.PipesK8sClient" class="hash-link"></a></Link></dt>
<dd>
A pipes client for launching kubernetes pods.
By default context is injected via environment variables and messages are parsed out of
the pod logs, with other logs forwarded to stdout of the orchestration process.
The first container within the containers list of the pod spec is expected (or set) to be
the container prepared for pipes protocol communication.
Parameters:
- <strong>env</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>str</em><em>]</em><em>]</em>) – An optional dict of environment variables to pass to the subprocess.
- <strong>context_injector</strong> (<em>Optional</em><em>[</em>[*PipesContextInjector*](../../../api/dagster/pipes.mdx#dagster.PipesContextInjector)<em>]</em>) – A context injector to use to inject context into the k8s container process. Defaults to `PipesEnvContextInjector`.
- <strong>message_reader</strong> (<em>Optional</em><em>[</em>[*PipesMessageReader*](../../../api/dagster/pipes.mdx#dagster.PipesMessageReader)<em>]</em>) – A message reader to use to read messages from the k8s container process. Defaults to [`PipesK8sPodLogsMessageReader`](#dagster_k8s.PipesK8sPodLogsMessageReader).
- <strong>load_incluster_config</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Whether this client is expected to be running from inside a kubernetes cluster and should load config using `kubernetes.config.load_incluster_config`. Otherwise `kubernetes.config.load_kube_config` is used with the kubeconfig_file argument. Default: None
- <strong>kubeconfig_file</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The value to pass as the config_file argument to `kubernetes.config.load_kube_config`. Default: None.
- <strong>kube_context</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The value to pass as the context argument to `kubernetes.config.load_kube_config`. Default: None.
- <strong>poll_interval</strong> (<em>Optional</em><em>[</em><em>float</em><em>]</em>) – How many seconds to wait between requests when polling the kubernetes API Default: 10.
<dl>
<dt><Link class="anchor" id='dagster_k8s.PipesK8sClient.run'>run<a href="#dagster_k8s.PipesK8sClient.run" class="hash-link"></a></Link></dt>
<dd>
Publish a kubernetes pod and wait for it to complete, enriched with the pipes protocol.
Parameters:
- <strong>context</strong> (<em>Union</em><em>[</em>[*OpExecutionContext*](../../../api/dagster/execution.mdx#dagster.OpExecutionContext)<em>, </em>[*AssetExecutionContext*](../../../api/dagster/execution.mdx#dagster.AssetExecutionContext)<em>]</em>) – The execution context.
- <strong>image</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The image to set the first container in the pod spec to use.
- <strong>command</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – The command to set the first container in the pod spec to use.
- <strong>namespace</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Which kubernetes namespace to use, defaults to the current namespace if running inside a kubernetes cluster or falling back to “default”.
- <strong>env</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>,</em><em>str</em><em>]</em><em>]</em>) – A mapping of environment variable names to values to set on the first container in the pod spec, on top of those configured on resource.
- <strong>base_pod_meta</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s pod’s metadata ([https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/object-meta/#ObjectMeta)) Keys can either snake_case or camelCase. The name value will be overridden.
- <strong>base_pod_spec</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – Raw k8s config for the k8s pod’s pod spec ([https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec)). Keys can either snake_case or camelCase. The dagster context will be readable from any container within the pod, but only the first container in the <cite>pod.spec.containers</cite> will be able to communicate back to Dagster.
- <strong>extras</strong> (<em>Optional</em><em>[</em><em>PipesExtras</em><em>]</em>) – Extra values to pass along as part of the ext protocol.
- <strong>ignore_containers</strong> (<em>Optional</em><em>[</em><em>Set</em><em>]</em>) – Ignore certain containers from waiting for termination. Defaults to None.
- <strong>enable_multi_container_logs</strong> (<em>bool</em>) – Whether or not to enable multi-container log consumption.
- <strong>pod_wait_timeout</strong> (<em>float</em>) – How long to wait for the pod to terminate before raising an exception. Defaults to 24h. Set to 0 to disable.
- <strong>delete_pod_on_completion</strong> (<em>bool</em>) – Whether to delete the pod after the run completes. Set to False to leave the pod in the cluster for debugging or to let the cluster handle pod deletion (e.g. via TTL or owner references). Defaults to True.
Returns:
Wrapper containing results reported by the external
process.
Return type: [PipesClientCompletedInvocation](../../../api/dagster/pipes.mdx#dagster.PipesClientCompletedInvocation)
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_k8s.PipesK8sPodLogsMessageReader'>`class` dagster_k8s.PipesK8sPodLogsMessageReader<a href="#dagster_k8s.PipesK8sPodLogsMessageReader" class="hash-link"></a></Link></dt>
<dd>
Message reader that reads messages from kubernetes pod logs.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-looker'
sidebar_position: 1000
title_meta: 'dagster-looker API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-looker Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-looker">
# dagster-looker
Dagster allows you to represent your Looker project as assets, alongside other your other
technologies like dbt and Sling. This allows you to see how your Looker assets are connected to
your other data assets, and how changes to other data assets might impact your Looker project.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerComponent'>`class` dagster_looker.LookerComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/components/looker_component.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerComponent" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Pulls in the contents of a Looker instance into Dagster assets.
Example:
```yaml
# defs.yaml
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: https://your-company.looker.com
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
looker_filter:
dashboard_folders:
- ["Shared"]
only_fetch_explores_used_in_dashboards: true
```
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/components/looker_component.py#L179' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Looker content item.
This method can be overridden in a subclass to customize how Looker content
(dashboards, looks, explores) are converted to Dagster asset specs. By default,
it delegates to the configured DagsterLookerApiTranslator.
Parameters: <strong>looker_structure</strong> – The LookerApiTranslatorStructureData containing information
about the Looker content item and instanceReturns: An AssetSpec that represents the Looker content as a Dagster asset
Example:
Override this method to add custom tags based on content properties:
```python
from dagster_looker import LookerComponent
from dagster import AssetSpec
class CustomLookerComponent(LookerComponent):
def get_asset_spec(self, looker_structure):
base_spec = super().get_asset_spec(looker_structure)
return base_spec.replace_attributes(
tags={
**base_spec.tags,
"looker_type": looker_structure.structure_data.structure_type,
"folder": looker_structure.structure_data.data.get("folder", {}).get("name")
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Looker component, see the [Looker component integration guide](https://docs.dagster.io/integrations/libraries/looker).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Looker component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: "{{ env.LOOKER_BASE_URL }}"
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
```
</div></div>
<div class="section" id="looker-api">
## Looker API
Here, we provide interfaces to manage Looker projects using the Looker API.
<div class="section" id="assets-looker-api">
### Assets (Looker API)
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerResource'>`class` dagster_looker.LookerResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/resource.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerResource" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Represents a connection to a Looker instance and provides methods
to interact with the Looker API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerApiTranslator'>`class` dagster_looker.DagsterLookerApiTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/dagster_looker_api_translator.py#L137' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerApiTranslator" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerApiTranslator.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/dagster_looker_api_translator.py#L254' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerApiTranslator.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Returns the `AssetSpec` representing a given Looker structure.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerStructureData'>`class` dagster_looker.LookerStructureData <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/dagster_looker_api_translator.py#L108' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerStructureData" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerStructureType'>`class` dagster_looker.LookerStructureType <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/dagster_looker_api_translator.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerStructureType" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.RequestStartPdtBuild'>`class` dagster_looker.RequestStartPdtBuild <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/dagster_looker_api_translator.py#L73' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.RequestStartPdtBuild" class="hash-link"></a></Link></dt>
<dd>
A request to start a PDT build. See [https://developers.looker.com/api/explorer/4.0/types/DerivedTable/RequestStartPdtBuild?sdk=py](https://developers.looker.com/api/explorer/4.0/types/DerivedTable/RequestStartPdtBuild?sdk=py)
for documentation on all available fields.
Parameters:
- <strong>model_name</strong> – The model of the PDT to start building.
- <strong>view_name</strong> – The view name of the PDT to start building.
- <strong>force_rebuild</strong> – Force rebuild of required dependent PDTs, even if they are already materialized.
- <strong>force_full_incremental</strong> – Force involved incremental PDTs to fully re-materialize.
- <strong>workspace</strong> – Workspace in which to materialize selected PDT (‘dev’ or default ‘production’).
- <strong>source</strong> – The source of this request.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.LookerFilter'>`class` dagster_looker.LookerFilter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/resource.py#L39' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.LookerFilter" class="hash-link"></a></Link></dt>
<dd>
Filters the set of Looker objects to fetch.
Parameters:
- <strong>dashboard_folders</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – A list of folder paths to fetch dashboards from. Each folder path is a list of folder names, starting from the root folder. All dashboards contained in the specified folders will be fetched. If not provided, all dashboards will be fetched.
- <strong>only_fetch_explores_used_in_dashboards</strong> (<em>bool</em>) – If True, only explores used in the fetched dashboards will be fetched. If False, all explores will be fetched. Defaults to False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.load_looker_asset_specs'>dagster_looker.load_looker_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/resource.py#L82' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.load_looker_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns a list of AssetSpecs representing the Looker structures.
Parameters:
- <strong>looker_resource</strong> ([*LookerResource*](#dagster_looker.LookerResource)) – The Looker resource to fetch assets from.
- <strong>dagster_looker_translator</strong> (<em>Optional</em><em>[</em>[*DagsterLookerApiTranslator*](#dagster_looker.DagsterLookerApiTranslator)<em>]</em>) – The translator to use to convert Looker structures into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterLookerApiTranslator`](#dagster_looker.DagsterLookerApiTranslator).
Returns: The set of AssetSpecs representing the Looker structures.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.build_looker_pdt_assets_definitions'>dagster_looker.build_looker_pdt_assets_definitions <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/api/assets.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.build_looker_pdt_assets_definitions" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns the AssetsDefinitions of the executable assets for the given the list of refreshable PDTs.
Parameters:
- <strong>resource_key</strong> (<em>str</em>) – The resource key to use for the Looker resource.
- <strong>request_start_pdt_builds</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em>[*RequestStartPdtBuild*](#dagster_looker.RequestStartPdtBuild)<em>]</em><em>]</em>) – A list of requests to start PDT builds. See [https://developers.looker.com/api/explorer/4.0/types/DerivedTable/RequestStartPdtBuild?sdk=py](https://developers.looker.com/api/explorer/4.0/types/DerivedTable/RequestStartPdtBuild?sdk=py) for documentation on all available fields.
- <strong>dagster_looker_translator</strong> (<em>Optional</em><em>[</em>[*DagsterLookerApiTranslator*](#dagster_looker.DagsterLookerApiTranslator)<em>]</em>) – The translator to use to convert Looker structures into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterLookerApiTranslator`](#dagster_looker.DagsterLookerApiTranslator).
Returns: The AssetsDefinitions of the executable assets for the given the list of refreshable PDTs.Return type: [AssetsDefinition](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)
</dd>
</dl>
</div></div>
<div class="section" id="lkml-lookml">
## lkml (LookML)
Here, we provide interfaces to manage Looker projects defined a set of locally accessible
LookML files.
<div class="section" id="assets-lkml">
### Assets (lkml)
<dl>
<dt><Link class="anchor" id='dagster_looker.build_looker_asset_specs'>dagster_looker.build_looker_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/asset_specs.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.build_looker_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Build a list of asset specs from a set of Looker structures defined in a Looker project.
Parameters:
- <strong>project_dir</strong> (<em>Path</em>) – The path to the Looker project directory.
- <strong>dagster_looker_translator</strong> (<em>Optional</em><em>[</em><em>DagsterLookerTranslator</em><em>]</em>) – Allows customizing how to map looker structures to asset keys and asset metadata.
Examples:
```python
from pathlib import Path
from dagster import Definitions
from dagster_looker import build_looker_asset_specs
looker_specs = build_looker_asset_specs(project_dir=Path("my_looker_project"))
defs = Definitions(assets=looker_specs)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator'>`class` dagster_looker.DagsterLookerLkmlTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L175' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Holds a set of methods that derive Dagster asset definition metadata given a representation
of a LookML structure (dashboards, explores, views).
This class is exposed so that methods can be overridden to customize how Dagster asset metadata
is derived.
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_asset_key'>get_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L254' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_asset_key" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).key` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster asset key that represents the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide a custom asset key for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: The Dagster asset key that represents the LookML structure.Return type: [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L184' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster asset spec that represents the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide a custom asset spec for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: The Dagster asset spec that represents the LookML structure.Return type: [AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_deps'>get_deps <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L338' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_deps" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Iterate over `DagsterLookerLkmlTranslator.get_asset_spec(...).deps` to access `AssetDep.asset_key` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster dependencies of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide custom dependencies for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: The Dagster dependencies for the LookML structure.Return type: Sequence[[AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_description'>get_description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L419' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_description" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).description` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster description of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide a custom description for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: The Dagster description for the LookML structure.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_group_name'>get_group_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L536' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_group_name" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).group_name` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster group name of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide a custom group name for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: A Dagster group name for the LookML structure.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_metadata'>get_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L477' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_metadata" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).metadata` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster metadata of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide custom metadata for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns:
A dictionary representing the Dagster metadata for the
LookML structure.
Return type: Optional[Mapping[str, Any]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_owners'>get_owners <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L592' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_owners" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).owners` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster owners of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide custom owners for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns: A sequence of Dagster owners for the LookML structure.Return type: Optional[Sequence[str]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_looker.DagsterLookerLkmlTranslator.get_tags'>get_tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-looker/dagster_looker/lkml/dagster_looker_lkml_translator.py#L648' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_looker.DagsterLookerLkmlTranslator.get_tags" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterLookerLkmlTranslator.get_asset_spec(...).tags` instead..
:::
A method that takes in a LookML structure (dashboards, explores, views) and
returns the Dagster tags of the structure.
The LookML structure is parsed using `lkml`. You can learn more about this here:
[https://lkml.readthedocs.io/en/latest/simple.html](https://lkml.readthedocs.io/en/latest/simple.html).
You can learn more about LookML dashboards and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/param-lookml-dashboard](https://cloud.google.com/looker/docs/reference/param-lookml-dashboard).
You can learn more about LookML explores and views and the properties available in this
dictionary here: [https://cloud.google.com/looker/docs/reference/lookml-quick-reference](https://cloud.google.com/looker/docs/reference/lookml-quick-reference).
This method can be overridden to provide custom tags for a LookML structure.
Parameters: <strong>lookml_structure</strong> (<em>Tuple</em><em>[</em><em>Path</em><em>, </em><em>str</em><em>, </em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – A tuple with the path to file
defining a LookML structure, the LookML structure type, and a dictionary
representing a LookML structure.Returns:
A dictionary representing the Dagster tags for the
LookML structure.
Return type: Optional[Mapping[str, str]]
</dd>
</dl>
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-mlflow library'
sidebar_position: 1000
title_meta: 'dagster-mlflow library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-mlflow library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-mlflow-library">
# dagster-mlflow library
<dl>
<dt><Link class="anchor" id='dagster_mlflow.mlflow_tracking'>dagster_mlflow.mlflow_tracking ResourceDefinition<a href="#dagster_mlflow.mlflow_tracking" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource initializes an MLflow run that’s used for all steps within a Dagster run.
This resource provides access to all of mlflow’s methods as well as the mlflow tracking client’s
methods.
Usage:
1. Add the mlflow resource to any ops in which you want to invoke mlflow tracking APIs.
2. Add the <cite>end_mlflow_on_run_finished</cite> hook to your job to end the MLflow run when the Dagster run is finished.
Examples:
```python
from dagster_mlflow import end_mlflow_on_run_finished, mlflow_tracking
@op(required_resource_keys={"mlflow"})
def mlflow_op(context):
mlflow.log_params(some_params)
mlflow.tracking.MlflowClient().create_registered_model(some_model_name)
@end_mlflow_on_run_finished
@job(resource_defs={"mlflow": mlflow_tracking})
def mlf_example():
mlflow_op()
# example using an mlflow instance with s3 storage
mlf_example.execute_in_process(run_config={
"resources": {
"mlflow": {
"config": {
"experiment_name": my_experiment,
"mlflow_tracking_uri": "http://localhost:5000",
# if want to run a nested run, provide parent_run_id
"parent_run_id": an_existing_mlflow_run_id,
# if you want to resume a run or avoid creating a new run in the resource init,
# provide mlflow_run_id
"mlflow_run_id": an_existing_mlflow_run_id,
# env variables to pass to mlflow
"env": {
"MLFLOW_S3_ENDPOINT_URL": my_s3_endpoint,
"AWS_ACCESS_KEY_ID": my_aws_key_id,
"AWS_SECRET_ACCESS_KEY": my_secret,
},
# env variables you want to log as mlflow tags
"env_to_tag": ["DOCKER_IMAGE_TAG"],
# key-value tags to add to your experiment
"extra_tags": {"super": "experiment"},
}
}
}
})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_mlflow.end_mlflow_on_run_finished'>dagster_mlflow.end_mlflow_on_run_finished HookDefinition<a href="#dagster_mlflow.end_mlflow_on_run_finished" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div>
---
---
title: 'dagster-duckdb-pandas library'
sidebar_position: 1000
title_meta: 'dagster-duckdb-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-duckdb-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-duckdb-pandas-library">
# dagster-duckdb-pandas library
This library provides an integration with the [DuckDB](https://duckdb.org) database and Pandas data processing library.
Related guides:
- [Using Dagster with DuckDB guide](https://docs.dagster.io/integrations/libraries/duckdb)
- [DuckDB I/O manager reference](https://docs.dagster.io/integrations/libraries/duckdb/reference)
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pandas.DuckDBPandasIOManager'>dagster_duckdb_pandas.DuckDBPandasIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-pandas/dagster_duckdb_pandas/duckdb_pandas_type_handler.py#L174' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pandas.DuckDBPandasIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes Pandas DataFrames to DuckDB. When
using the DuckDBPandasIOManager, any inputs and outputs without type annotations will be loaded
as Pandas DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_pandas import DuckDBPandasIOManager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPandasIOManager(database="my_db.duckdb")}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPandasIOManager(database="my_db.duckdb", schema="my_schema")}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pandas.DuckDBPandasTypeHandler'>`class` dagster_duckdb_pandas.DuckDBPandasTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-pandas/dagster_duckdb_pandas/duckdb_pandas_type_handler.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pandas.DuckDBPandasTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Stores and loads Pandas DataFrames in DuckDB.
To use this type handler, return it from the `type_handlers` method of an I/O manager that inherits from ``DuckDBIOManager`.
Example:
```python
from dagster_duckdb import DuckDBIOManager
from dagster_duckdb_pandas import DuckDBPandasTypeHandler
class MyDuckDBIOManager(DuckDBIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DuckDBPandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb")}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pandas.duckdb_pandas_io_manager'>dagster_duckdb_pandas.duckdb_pandas_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pandas.duckdb_pandas_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes Pandas DataFrames to DuckDB. When
using the duckdb_pandas_io_manager, any inputs and outputs without type annotations will be loaded
as Pandas DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_pandas import duckdb_pandas_io_manager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_pandas_io_manager.configured({"database": "my_db.duckdb"})}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_pandas_io_manager.configured({"database": "my_db.duckdb", "schema": "my_schema"})}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-duckdb-polars library'
sidebar_position: 1000
title_meta: 'dagster-duckdb-polars library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-duckdb-polars library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-duckdb-polars-library">
# dagster-duckdb-polars library
This library provides an integration with the [DuckDB](https://duckdb.org) database and Polars data processing library.
Related guides:
- [Using Dagster with DuckDB guide](https://docs.dagster.io/integrations/libraries/duckdb)
- [DuckDB I/O manager reference](https://docs.dagster.io/integrations/libraries/duckdb/reference)
<dl>
<dt><Link class="anchor" id='dagster_duckdb_polars.DuckDBPolarsIOManager'>dagster_duckdb_polars.DuckDBPolarsIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-polars/dagster_duckdb_polars/duckdb_polars_type_handler.py#L174' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_polars.DuckDBPolarsIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes Polars DataFrames to DuckDB. When
using the DuckDBPolarsIOManager, any inputs and outputs without type annotations will be loaded
as Polars DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_polars import DuckDBPolarsIOManager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPolarsIOManager(database="my_db.duckdb")}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPolarsIOManager(database="my_db.duckdb", schema="my_schema")}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pl.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pl.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pl.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
>
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame) -> pl.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_duckdb_polars.DuckDBPolarsTypeHandler'>`class` dagster_duckdb_polars.DuckDBPolarsTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-polars/dagster_duckdb_polars/duckdb_polars_type_handler.py#L9' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_polars.DuckDBPolarsTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Stores and loads Polars DataFrames in DuckDB.
To use this type handler, return it from the `type_handlers` method of an I/O manager that inherits from ``DuckDBIOManager`.
Example:
```python
from dagster_duckdb import DuckDBIOManager
from dagster_duckdb_polars import DuckDBPolarsTypeHandler
class MyDuckDBIOManager(DuckDBIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DuckDBPolarsTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb")}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_duckdb_polars.duckdb_polars_io_manager'>dagster_duckdb_polars.duckdb_polars_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_polars.duckdb_polars_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes polars dataframes to DuckDB. When
using the duckdb_polars_io_manager, any inputs and outputs without type annotations will be loaded
as Polars DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_polars import duckdb_polars_io_manager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_polars_io_manager.configured({"database": "my_db.duckdb"})}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_polars_io_manager.configured({"database": "my_db.duckdb", "schema": "my_schema"})}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pl.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pl.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pl.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame) -> pl.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-duckdb-pyspark library'
sidebar_position: 1000
title_meta: 'dagster-duckdb-pyspark library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-duckdb-pyspark library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-duckdb-pyspark-library">
# dagster-duckdb-pyspark library
This library provides an integration with the [DuckDB](https://duckdb.org) database and PySpark data processing library.
Related guides:
- [Using Dagster with DuckDB guide](https://docs.dagster.io/integrations/libraries/duckdb)
- [DuckDB I/O manager reference](https://docs.dagster.io/integrations/libraries/duckdb/reference)
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pyspark.DuckDBPySparkIOManager'>dagster_duckdb_pyspark.DuckDBPySparkIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-pyspark/dagster_duckdb_pyspark/duckdb_pyspark_type_handler.py#L183' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pyspark.DuckDBPySparkIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to DuckDB. When
using the DuckDBPySparkIOManager, any inputs and outputs without type annotations will be loaded
as PySpark DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_pyspark import DuckDBPySparkIOManager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pyspark.sql.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPySparkIOManager(database="my_db.duckdb")}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": DuckDBPySparkIOManager(database="my_db.duckdb", schema="my_schema")}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pyspark.sql.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pyspark.sql.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pyspark.sql.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pyspark.DuckDBPySparkTypeHandler'>`class` dagster_duckdb_pyspark.DuckDBPySparkTypeHandler <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb-pyspark/dagster_duckdb_pyspark/duckdb_pyspark_type_handler.py#L19' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pyspark.DuckDBPySparkTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Stores PySpark DataFrames in DuckDB.
To use this type handler, return it from the `type_handlers` method of an I/O manager that inherits from ``DuckDBIOManager`.
Example:
```python
from dagster_duckdb import DuckDBIOManager
from dagster_duckdb_pyspark import DuckDBPySparkTypeHandler
class MyDuckDBIOManager(DuckDBIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DuckDBPySparkTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pyspark.sql.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb")}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_duckdb_pyspark.duckdb_pyspark_io_manager'>dagster_duckdb_pyspark.duckdb_pyspark_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L116' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb_pyspark.duckdb_pyspark_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to DuckDB. When
using the duckdb_pyspark_io_manager, any inputs and outputs without type annotations will be loaded
as PySpark DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb_pyspark import duckdb_pyspark_io_manager
@asset(
key_prefix=["my_schema"] # will be used as the schema in DuckDB
)
def my_table() -> pyspark.sql.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_pyspark_io_manager.configured({"database": "my_db.duckdb"})}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": duckdb_pyspark_io_manager.configured({"database": "my_db.duckdb", "schema": "my_schema"})}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pyspark.sql.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pyspark.sql.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pyspark.sql.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pyspark.sql.DataFrame) -> pyspark.sql.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-duckdb library'
sidebar_position: 1000
title_meta: 'dagster-duckdb library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-duckdb library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-duckdb-library">
# dagster-duckdb library
This library provides an integration with the [DuckDB](https://duckdb.org) database.
Related Guides:
- [Using Dagster with DuckDB guide](https://docs.dagster.io/integrations/libraries/duckdb)
- [DuckDB I/O manager reference](https://docs.dagster.io/integrations/libraries/duckdb/reference)
<dl>
<dt><Link class="anchor" id='dagster_duckdb.DuckDBIOManager'>dagster_duckdb.DuckDBIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L138' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb.DuckDBIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to DuckDB.
Examples:
```python
from dagster_duckdb import DuckDBIOManager
from dagster_duckdb_pandas import DuckDBPandasTypeHandler
class MyDuckDBIOManager(DuckDBIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [DuckDBPandasTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb")}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb", schema="my_schema")}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
Set DuckDB configuration options using the connection_config field. See
[https://duckdb.org/docs/sql/configuration.html](https://duckdb.org/docs/sql/configuration.html) for all available settings.
```python
Definitions(
assets=[my_table],
resources={"io_manager": MyDuckDBIOManager(database="my_db.duckdb",
connection_config={"arrow_large_buffer_size": True})}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_duckdb.DuckDBResource'>dagster_duckdb.DuckDBResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/resource.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb.DuckDBResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with a DuckDB database.
Examples:
```python
from dagster import Definitions, asset
from dagster_duckdb import DuckDBResource
@asset
def my_table(duckdb: DuckDBResource):
with duckdb.get_connection() as conn:
conn.execute("SELECT * from MY_SCHEMA.MY_TABLE")
Definitions(
assets=[my_table],
resources={"duckdb": DuckDBResource(database="path/to/db.duckdb")}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_duckdb.build_duckdb_io_manager'>dagster_duckdb.build_duckdb_io_manager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-duckdb/dagster_duckdb/io_manager.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_duckdb.build_duckdb_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Builds an IO manager definition that reads inputs from and writes outputs to DuckDB.
Parameters:
- <strong>type_handlers</strong> (<em>Sequence</em><em>[</em><em>DbTypeHandler</em><em>]</em>) – Each handler defines how to translate between DuckDB tables and an in-memory type - e.g. a Pandas DataFrame. If only one DbTypeHandler is provided, it will be used as the default_load_type.
- <strong>default_load_type</strong> (<em>Type</em>) – When an input has no type annotation, load it as this type.
Returns: IOManagerDefinition
Examples:
```python
from dagster_duckdb import build_duckdb_io_manager
from dagster_duckdb_pandas import DuckDBPandasTypeHandler
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
duckdb_io_manager = build_duckdb_io_manager([DuckDBPandasTypeHandler()])
Definitions(
assets=[my_table]
resources={"io_manager" duckdb_io_manager.configured({"database": "my_db.duckdb"})}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the DuckDB I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table]
resources={"io_manager" duckdb_io_manager.configured(
{"database": "my_db.duckdb", "schema": "my_schema"} # will be used as the schema
)}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in duckdb
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in duckdb
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame):
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-msteams library'
sidebar_position: 1000
title_meta: 'dagster-msteams library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-msteams library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-msteams-library">
# dagster-msteams library
<div class="section" id="resource">
## Resource
<dl>
<dt><Link class="anchor" id='dagster_msteams.MSTeamsResource'>dagster_msteams.MSTeamsResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-msteams/dagster_msteams/resources.py#L8' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_msteams.MSTeamsResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource is for connecting to Microsoft Teams.
Provides a <cite>dagster_msteams.TeamsClient</cite> which can be used to
interface with the MS Teams API.
By configuring this resource, you can post messages to MS Teams from any Dagster op,
asset, schedule, or sensor:
Examples:
```python
import os
from dagster import op, job, Definitions, EnvVar
from dagster_msteams import Card, MSTeamsResource
@op
def teams_op(msteams: MSTeamsResource):
card = Card()
card.add_attachment(text_message="Hello There !!")
msteams.get_client().post_message(payload=card.payload)
@job
def teams_job():
teams_op()
Definitions(
jobs=[teams_job],
resources={
"msteams": MSTeamsResource(
hook_url=EnvVar("TEAMS_WEBHOOK_URL")
)
}
)
```
</dd>
</dl>
</div>
<div class="section" id="sensors">
## Sensors
<dl>
<dt><Link class="anchor" id='dagster_msteams.teams_on_failure'>dagster_msteams.teams_on_failure HookDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-msteams/dagster_msteams/hooks.py#L26' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_msteams.teams_on_failure" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step failure events that will message the given MS Teams webhook URL.
Parameters:
- <strong>message_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*HookContext*](../../../api/dagster/hooks.mdx#dagster.HookContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the HookContext outputs the message you want to send.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
Examples:
```python
@teams_on_failure(webserver_base_url="http://localhost:3000")
@job(...)
def my_job():
pass
```
```python
def my_message_fn(context: HookContext) -> str:
return f"Op {context.op.name} failed!"
@op
def a_op(context):
pass
@job(...)
def my_job():
a_op.with_hooks(hook_defs={teams_on_failure("#foo", my_message_fn)})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_msteams.teams_on_success'>dagster_msteams.teams_on_success HookDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-msteams/dagster_msteams/hooks.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_msteams.teams_on_success" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step success events that will message the given MS Teams webhook URL.
Parameters:
- <strong>message_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*HookContext*](../../../api/dagster/hooks.mdx#dagster.HookContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the HookContext outputs the message you want to send.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
Examples:
```python
@teams_on_success(webserver_base_url="http://localhost:3000")
@job(...)
def my_job():
pass
```
```python
def my_message_fn(context: HookContext) -> str:
return f"Op {context.op.name} failed!"
@op
def a_op(context):
pass
@job(...)
def my_job():
a_op.with_hooks(hook_defs={teams_on_success("#foo", my_message_fn)})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_msteams.make_teams_on_run_failure_sensor'>dagster_msteams.make_teams_on_run_failure_sensor <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-msteams/dagster_msteams/sensors.py#L33' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_msteams.make_teams_on_run_failure_sensor" class="hash-link"></a></Link></dt>
<dd>
Create a sensor on run failures that will message the given MS Teams webhook URL.
Parameters:
- <strong>hook_url</strong> (<em>str</em>) – MS Teams incoming webhook URL.
- <strong>message_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*RunFailureSensorContext*](../../../api/dagster/schedules-sensors.mdx#dagster.RunFailureSensorContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the `RunFailureSensorContext` and outputs the message you want to send. Defaults to a text message that contains error message, job name, and run ID.
- <strong>http_proxy</strong> – (Optional[str]): Proxy for requests using http protocol.
- <strong>https_proxy</strong> – (Optional[str]): Proxy for requests using https protocol.
- <strong>timeout</strong> – (Optional[float]): Connection timeout in seconds. Defaults to 60.
- <strong>verify</strong> – (Optional[bool]): Whether to verify the servers TLS certificate.
- <strong>name</strong> – (Optional[str]): The name of the sensor. Defaults to “teams_on_run_failure”.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the failed run.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from Dagit or via the GraphQL API.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](../../../api/dagster/jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](../../../api/dagster/graphs.mdx#dagster.GraphDefinition)<em>, </em><em>UnresolvedAssetJobDefinition</em><em>, </em>[*RepositorySelector*](../../../api/dagster/schedules-sensors.mdx#dagster.RepositorySelector)<em>, </em>[*JobSelector*](../../../api/dagster/schedules-sensors.mdx#dagster.JobSelector)<em>]</em><em>]</em><em>]</em>) – Jobs in the current repository that will be monitored by this sensor. Defaults to None, which means the alert will be sent when any job in the repository matches the requested run_status. To monitor jobs in external repositories, use RepositorySelector and JobSelector.
- <strong>monitor_all_code_locations</strong> (<em>bool</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the failed run.
- <strong>monitor_all_repositories</strong> (<em>bool</em>) – <span className="flag flag-warning">deprecated</span> If set to True, the sensor will monitor all runs in the Dagster instance. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
Examples:
```python
teams_on_run_failure = make_teams_on_run_failure_sensor(
hook_url=os.getenv("TEAMS_WEBHOOK_URL")
)
@repository
def my_repo():
return [my_job + teams_on_run_failure]
```
```python
def my_message_fn(context: RunFailureSensorContext) -> str:
return "Job {job_name} failed! Error: {error}".format(
job_name=context.dagster_run.job_name,
error=context.failure_event.message,
)
teams_on_run_failure = make_teams_on_run_failure_sensor(
hook_url=os.getenv("TEAMS_WEBHOOK_URL"),
message_fn=my_message_fn,
webserver_base_url="http://localhost:3000",
)
```
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_msteams.msteams_resource'>dagster_msteams.msteams_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-msteams/dagster_msteams/resources.py#L77' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_msteams.msteams_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource is for connecting to Microsoft Teams.
The resource object is a <cite>dagster_msteams.TeamsClient</cite>.
By configuring this resource, you can post messages to MS Teams from any Dagster solid:
Examples:
```python
import os
from dagster import op, job
from dagster_msteams import Card, msteams_resource
@op(required_resource_keys={"msteams"})
def teams_op(context):
card = Card()
card.add_attachment(text_message="Hello There !!")
context.resources.msteams.post_message(payload=card.payload)
@job(resource_defs={"msteams": msteams_resource})
def teams_job():
teams_op()
teams_job.execute_in_process(
{"resources": {"msteams": {"config": {"hook_url": os.getenv("TEAMS_WEBHOOK_URL")}}}}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-mysql library'
sidebar_position: 1000
title_meta: 'dagster-mysql library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-mysql library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-mysql-library">
# dagster-mysql library
<dl>
<dt><Link class="anchor" id='dagster_mysql.MySQLResource'>dagster_mysql.MySQLResource ResourceDefinition<a href="#dagster_mysql.MySQLResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Resource for interacting with a MySQL database. Wraps an underlying mysql.connector connection.
Examples:
```python
from dagster import Definitions, asset, EnvVar
from dagster_mysql import MySQLResource
@asset
def my_table(mysql: MySQLResource):
with mysql.get_connection() as conn:
with conn.cursor() as cur:
cur.execute("SELECT * FROM table;")
results = cur.fetchall()
defs = Definitions(
assets=[my_table],
resources={
"mysql": MySQLResource(
host="localhost",
port=3306,
user="root",
password=EnvVar("MYSQL_PASSWORD")
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_mysql.MySQLEventLogStorage'>`class` dagster_mysql.MySQLEventLogStorage<a href="#dagster_mysql.MySQLEventLogStorage" class="hash-link"></a></Link></dt>
<dd>
MySQL-backed event log storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
dagster.yaml
```YAML
event_log_storage:
module: dagster_mysql.event_log
class: MySQLEventLogStorage
config:
mysql_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { db_name }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_mysql.MySQLRunStorage'>`class` dagster_mysql.MySQLRunStorage<a href="#dagster_mysql.MySQLRunStorage" class="hash-link"></a></Link></dt>
<dd>
MySQL-backed run storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
dagster.yaml
```YAML
run_storage:
module: dagster_mysql.run_storage
class: MySQLRunStorage
config:
mysql_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { database }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_mysql.MySQLScheduleStorage'>`class` dagster_mysql.MySQLScheduleStorage<a href="#dagster_mysql.MySQLScheduleStorage" class="hash-link"></a></Link></dt>
<dd>
MySQL-backed run storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
dagster.yaml
```YAML
schedule_storage:
module: dagster_mysql.schedule_storage
class: MySQLScheduleStorage
config:
mysql_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { db_name }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
</div>
---
---
title: 'dagster-omni library'
sidebar_position: 1000
title_meta: 'dagster-omni library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-omni library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-omni-library">
# dagster-omni library
Dagster allows you to represent your Omni documents as assets, with dependencies on the data
assets (e.g. database tables) that they depend on. This allows you to understand how changes to
upstream data may interact with end product dashboards.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_omni.OmniComponent'>`class` dagster_omni.OmniComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-omni/dagster_omni/component.py#L40' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_omni.OmniComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Pulls in the contents of an Omni workspace into Dagster assets.
Example:
```yaml
# defs.yaml
type: dagster_omni.OmniComponent
attributes:
workspace:
base_url: https://your-company.omniapp.co
api_key: "{{ env.OMNI_API_KEY }}"
```
<dl>
<dt><Link class="anchor" id='dagster_omni.OmniComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-omni/dagster_omni/component.py#L117' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_omni.OmniComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Omni document.
This method can be overridden in a subclass to customize how Omni documents
(workbooks, queries) are converted to Dagster asset specs. By default, it applies
any configured translation function to the base asset spec.
Parameters:
- <strong>context</strong> – The component load context provided by Dagster
- <strong>data</strong> – The OmniTranslatorData containing information about the Omni document
Returns: An AssetSpec that represents the Omni document as a Dagster asset, or None
if the document should not be represented as an asset
Example:
Override this method to add custom metadata based on document properties:
```python
from dagster_omni import OmniComponent
import dagster as dg
class CustomOmniComponent(OmniComponent):
def get_asset_spec(self, context, data):
base_spec = super().get_asset_spec(context, data)
if base_spec:
return base_spec.replace_attributes(
metadata={
**base_spec.metadata,
"omni_type": type(data.obj).__name__,
"workspace": data.workspace_data.workspace_id
}
)
return None
```
</dd>
</dl>
</dd>
</dl>
The main class for interacting with Omni is the `OmniComponent`. This class is responsible for connecting to your Omni instance,
fetching information about your documents, and building Dagster asset definitions from that information.
`OmniComponent` is a `StateBackedComponent`, which means that it only fetches updated information from the Omni API when you tell
it to, and you will need to redeploy your code location after updating your metadata in order to see those changes.
The simplest way to update the stored state of your `OmniComponent` is to use the `dg utils refresh-component-state` command. When
deploying your code location, this command should be executed in your CI/CD workflow (e.g. github actions).
</div></div>
---
---
title: 'dagster-openai library'
sidebar_position: 1000
title_meta: 'dagster-openai library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-openai library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-openai-library">
# dagster-openai library
The <cite>dagster_openai</cite> library provides utilities for using OpenAI with Dagster.
A good place to start with <cite>dagster_openai</cite> is [the guide](https://docs.dagster.io/integrations/libraries/openai).
<dl>
<dt><Link class="anchor" id='dagster_openai.with_usage_metadata'>dagster_openai.with_usage_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-openai/dagster_openai/resources.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_openai.with_usage_metadata" class="hash-link"></a></Link></dt>
<dd>
This wrapper can be used on any endpoint of the
[openai library](https://github.com/openai/openai-python)
to log the OpenAI API usage metadata in the asset metadata.
Examples:
```python
from dagster import (
AssetExecutionContext,
AssetKey,
AssetSelection,
AssetSpec,
Definitions,
EnvVar,
MaterializeResult,
asset,
define_asset_job,
multi_asset,
)
from dagster_openai import OpenAIResource, with_usage_metadata
@asset(compute_kind="OpenAI")
def openai_asset(context: AssetExecutionContext, openai: OpenAIResource):
with openai.get_client(context) as client:
client.fine_tuning.jobs.create = with_usage_metadata(
context=context, output_name="some_output_name", func=client.fine_tuning.jobs.create
)
client.fine_tuning.jobs.create(model="gpt-3.5-turbo", training_file="some_training_file")
openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")
@multi_asset(
specs=[
AssetSpec("my_asset1"),
AssetSpec("my_asset2"),
]
)
def openai_multi_asset(context: AssetExecutionContext, openai: OpenAIResource):
with openai.get_client(context, asset_key=AssetKey("my_asset1")) as client:
client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}]
)
# The materialization of `my_asset1` will include both OpenAI usage metadata
# and the metadata added when calling `MaterializeResult`.
return (
MaterializeResult(asset_key="my_asset1", metadata={"foo": "bar"}),
MaterializeResult(asset_key="my_asset2", metadata={"baz": "qux"}),
)
openai_multi_asset_job = define_asset_job(
name="openai_multi_asset_job", selection=AssetSelection.assets(openai_multi_asset)
)
Definitions(
assets=[openai_asset, openai_multi_asset],
jobs=[openai_asset_job, openai_multi_asset_job],
resources={
"openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")),
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_openai.OpenAIResource'>`class` dagster_openai.OpenAIResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-openai/dagster_openai/resources.py#L160' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_openai.OpenAIResource" class="hash-link"></a></Link></dt>
<dd>
This resource is wrapper over the
[openai library](https://github.com/openai/openai-python).
By configuring this OpenAI resource, you can interact with OpenAI API
and log its usage metadata in the asset metadata.
Examples:
```python
import os
from dagster import AssetExecutionContext, Definitions, EnvVar, asset, define_asset_job
from dagster_openai import OpenAIResource
@asset(compute_kind="OpenAI")
def openai_asset(context: AssetExecutionContext, openai: OpenAIResource):
with openai.get_client(context) as client:
client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Say this is a test"}]
)
openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")
Definitions(
assets=[openai_asset],
jobs=[openai_asset_job],
resources={
"openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")),
},
)
```
<dl>
<dt><Link class="anchor" id='dagster_openai.OpenAIResource.get_client'>get_client <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-openai/dagster_openai/resources.py#L239' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_openai.OpenAIResource.get_client" class="hash-link"></a></Link></dt>
<dd>
Yields an `openai.Client` for interacting with the OpenAI API.
By default, in an asset context, the client comes with wrapped endpoints
for three API resources, Completions, Embeddings and Chat,
allowing to log the API usage metadata in the asset metadata.
Note that the endpoints are not and cannot be wrapped
to automatically capture the API usage metadata in an op context.
Parameters: <strong>context</strong> – The `context` object for computing the op or asset in which `get_client` is called.
Examples:
```python
from dagster import (
AssetExecutionContext,
Definitions,
EnvVar,
GraphDefinition,
OpExecutionContext,
asset,
define_asset_job,
op,
)
from dagster_openai import OpenAIResource
@op
def openai_op(context: OpExecutionContext, openai: OpenAIResource):
with openai.get_client(context) as client:
client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}]
)
openai_op_job = GraphDefinition(name="openai_op_job", node_defs=[openai_op]).to_job()
@asset(compute_kind="OpenAI")
def openai_asset(context: AssetExecutionContext, openai: OpenAIResource):
with openai.get_client(context) as client:
client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}]
)
openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")
Definitions(
assets=[openai_asset],
jobs=[openai_asset_job, openai_op_job],
resources={
"openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")),
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_openai.OpenAIResource.get_client_for_asset'>get_client_for_asset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-openai/dagster_openai/resources.py#L302' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_openai.OpenAIResource.get_client_for_asset" class="hash-link"></a></Link></dt>
<dd>
Yields an `openai.Client` for interacting with the OpenAI.
When using this method, the OpenAI API usage metadata is automatically
logged in the asset materializations associated with the provided `asset_key`.
By default, the client comes with wrapped endpoints
for three API resources, Completions, Embeddings and Chat,
allowing to log the API usage metadata in the asset metadata.
This method can only be called when working with assets,
i.e. the provided `context` must be of type `AssetExecutionContext`.
Parameters:
- <strong>context</strong> – The `context` object for computing the asset in which `get_client` is called.
- <strong>asset_key</strong> – the `asset_key` of the asset for which a materialization should include the metadata.
Examples:
```python
from dagster import (
AssetExecutionContext,
AssetKey,
AssetSpec,
Definitions,
EnvVar,
MaterializeResult,
asset,
define_asset_job,
multi_asset,
)
from dagster_openai import OpenAIResource
@asset(compute_kind="OpenAI")
def openai_asset(context: AssetExecutionContext, openai: OpenAIResource):
with openai.get_client_for_asset(context, context.asset_key) as client:
client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}]
)
openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")
@multi_asset(specs=[AssetSpec("my_asset1"), AssetSpec("my_asset2")], compute_kind="OpenAI")
def openai_multi_asset(context: AssetExecutionContext, openai_resource: OpenAIResource):
with openai_resource.get_client_for_asset(context, asset_key=AssetKey("my_asset1")) as client:
client.chat.completions.create(
model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}]
)
return (
MaterializeResult(asset_key="my_asset1", metadata={"some_key": "some_value1"}),
MaterializeResult(asset_key="my_asset2", metadata={"some_key": "some_value2"}),
)
openai_multi_asset_job = define_asset_job(
name="openai_multi_asset_job", selection="openai_multi_asset"
)
Definitions(
assets=[openai_asset, openai_multi_asset],
jobs=[openai_asset_job, openai_multi_asset_job],
resources={
"openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")),
},
)
```
</dd>
</dl>
</dd>
</dl>
</div>
---
---
title: 'dagster-pagerduty library'
sidebar_position: 1000
title_meta: 'dagster-pagerduty library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-pagerduty library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pagerduty-library">
# dagster-pagerduty library
This library provides an integration with PagerDuty, to support creating alerts from your Dagster
code.
Presently, it provides a thin wrapper on the [Events API V2](https://v2.developer.pagerduty.com/docs/events-api-v2).
<div class="section" id="getting-started">
## Getting started
You can install this library with:
```default
pip install dagster-pagerduty
```
To use this integration, you’ll first need to create an Events API V2 PagerDuty integration on a PagerDuty service. There are instructions
[here](https://support.pagerduty.com/docs/services-and-integrations#section-events-api-v2) for
creating a new PagerDuty service & integration.
Once your Events API V2 integration is set up, you’ll find an Integration Key (also referred to as a
“Routing Key”) on the Integrations tab for your service. This key is used to authorize events
created from the PagerDuty events API.
Once your service/integration is created, you can provision a PagerDuty resource and issue PagerDuty
alerts from within your ops.
<dl>
<dt><Link class="anchor" id='dagster_pagerduty.PagerDutyService'>dagster_pagerduty.PagerDutyService ResourceDefinition<a href="#dagster_pagerduty.PagerDutyService" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource is for posting events to PagerDuty.
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_pagerduty.pagerduty_resource'>dagster_pagerduty.pagerduty_resource ResourceDefinition<a href="#dagster_pagerduty.pagerduty_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
A resource for posting events (alerts) to PagerDuty.
Example:
```python
@op
def pagerduty_op(pagerduty: PagerDutyService):
pagerduty.EventV2_create(
summary='alert from dagster',
source='localhost',
severity='error',
event_action='trigger',
)
@job(resource_defs={ 'pagerduty': pagerduty_resource })
def pagerduty_test():
pagerduty_op()
pagerduty_test.execute_in_process(
run_config={
"resources": {
'pagerduty': {'config': {'routing_key': '0123456789abcdef0123456789abcdef'}}
}
}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-pandas library'
sidebar_position: 1000
title_meta: 'dagster-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pandas-library">
# dagster-pandas library
The <cite>dagster_pandas</cite> library provides utilities for using pandas with Dagster and for implementing
validation on pandas <cite>DataFrames</cite>. A good place to start with <cite>dagster_pandas</cite> is the [validation
guide](https://docs.dagster.io/integrations/libraries/pandas).
<dl>
<dt><Link class="anchor" id='dagster_pandas.create_dagster_pandas_dataframe_type'>dagster_pandas.create_dagster_pandas_dataframe_type <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py#L138' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pandas.create_dagster_pandas_dataframe_type" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Constructs a custom pandas dataframe dagster type.
Parameters:
- <strong>name</strong> (<em>str</em>) – Name of the dagster pandas type.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A markdown-formatted string, displayed in tooling.
- <strong>columns</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em>[*PandasColumn*](#dagster_pandas.PandasColumn)<em>]</em><em>]</em>) – A list of `PandasColumn` objects which express dataframe column schemas and constraints.
- <strong>metadata_fn</strong> (<em>Optional</em><em>[</em><em>Callable</em><em>[</em><em>[</em><em>]</em><em>, </em><em>Union</em><em>[</em><em>Dict</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>str</em><em>, </em><em>float</em><em>, </em><em>int</em><em>, </em><em>Dict</em><em>, </em>[*MetadataValue*](../../../api/dagster/metadata.mdx#dagster.MetadataValue)<em>]</em><em>]</em>) – A callable which takes your dataframe and returns a dict with string label keys and MetadataValue values.
- <strong>dataframe_constraints</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>DataFrameConstraint</em><em>]</em><em>]</em>) – A list of objects that inherit from `DataFrameConstraint`. This allows you to express dataframe-level constraints.
- <strong>loader</strong> (<em>Optional</em><em>[</em>[*DagsterTypeLoader*](../../../api/dagster/types.mdx#dagster.DagsterTypeLoader)<em>]</em>) – An instance of a class that inherits from [`DagsterTypeLoader`](../../../api/dagster/types.mdx#dagster.DagsterTypeLoader). If None, we will default to using <cite>dataframe_loader</cite>.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pandas.RowCountConstraint'>`class` dagster_pandas.RowCountConstraint <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pandas/dagster_pandas/constraints.py#L337' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pandas.RowCountConstraint" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A dataframe constraint that validates the expected count of rows.
Parameters:
- <strong>num_allowed_rows</strong> (<em>int</em>) – The number of allowed rows in your dataframe.
- <strong>error_tolerance</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The acceptable threshold if you are not completely certain. Defaults to 0.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pandas.StrictColumnsConstraint'>`class` dagster_pandas.StrictColumnsConstraint <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pandas/dagster_pandas/constraints.py#L297' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pandas.StrictColumnsConstraint" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A dataframe constraint that validates column existence and ordering.
Parameters:
- <strong>strict_column_list</strong> (<em>List</em><em>[</em><em>str</em><em>]</em>) – The exact list of columns that your dataframe must have.
- <strong>enforce_ordering</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If true, will enforce that the ordering of column names must match. Default is False.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pandas.PandasColumn'>`class` dagster_pandas.PandasColumn <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pandas/dagster_pandas/validation.py#L46' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pandas.PandasColumn" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
The main API for expressing column level schemas and constraints for your custom dataframe
types.
Parameters:
- <strong>name</strong> (<em>str</em>) – Name of the column. This must match up with the column name in the dataframe you expect to receive.
- <strong>is_required</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – Flag indicating the optional/required presence of the column. If th column exists, the validate function will validate the column. Defaults to True.
- <strong>constraints</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Constraint</em><em>]</em><em>]</em>) – List of constraint objects that indicate the validation rules for the pandas column.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pandas.DataFrame'>dagster_pandas.DataFrame `=` \<dagster._core.types.dagster_type.DagsterType object><a href="#dagster_pandas.DataFrame" class="hash-link"></a></Link></dt>
<dd>
Define a type in dagster. These can be used in the inputs and outputs of ops.
Parameters:
- <strong>type_check_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*TypeCheckContext*](../../../api/dagster/execution.mdx#dagster.TypeCheckContext)<em>, </em><em>Any</em><em>]</em><em>, </em><em>[</em><em>Union</em><em>[</em><em>bool</em><em>, </em>[*TypeCheck*](../../../api/dagster/ops.mdx#dagster.TypeCheck)<em>]</em><em>]</em><em>]</em>) – The function that defines the type check. It takes the value flowing through the input or output of the op. If it passes, return either `True` or a [`TypeCheck`](../../../api/dagster/ops.mdx#dagster.TypeCheck) with `success` set to `True`. If it fails, return either `False` or a [`TypeCheck`](../../../api/dagster/ops.mdx#dagster.TypeCheck) with `success` set to `False`. The first argument must be named `context` (or, if unused, `_`, `_context`, or `context_`). Use `required_resource_keys` for access to resources.
- <strong>key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) –
The unique key to identify types programmatically. The key property always has a value. If you omit key to the argument to the init function, it instead receives the value of `name`. If neither `key` nor `name` is provided, a `CheckError` is thrown.
In the case of a generic type such as `List` or `Optional`, this is generated programmatically based on the type parameters.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A unique name given by a user. If `key` is `None`, `key` becomes this value. Name is not given in a case where the user does not specify a unique name for this type, such as a generic class.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A markdown-formatted string, displayed in tooling.
- <strong>loader</strong> (<em>Optional</em><em>[</em>[*DagsterTypeLoader*](../../../api/dagster/types.mdx#dagster.DagsterTypeLoader)<em>]</em>) – An instance of a class that inherits from [`DagsterTypeLoader`](../../../api/dagster/types.mdx#dagster.DagsterTypeLoader) and can map config data to a value of this type. Specify this argument if you will need to shim values of this type using the config machinery. As a rule, you should use the [`@dagster_type_loader`](../../../api/dagster/types.mdx#dagster.dagster_type_loader) decorator to construct these arguments.
- <strong>required_resource_keys</strong> (<em>Optional</em><em>[</em><em>Set</em><em>[</em><em>str</em><em>]</em><em>]</em>) – Resource keys required by the `type_check_fn`.
- <strong>is_builtin</strong> (<em>bool</em>) – Defaults to False. This is used by tools to display or filter built-in types (such as `String`, `Int`) to visually distinguish them from user-defined types. Meant for internal use.
- <strong>kind</strong> (<em>DagsterTypeKind</em>) – Defaults to None. This is used to determine the kind of runtime type for InputDefinition and OutputDefinition type checking.
- <strong>typing_type</strong> – Defaults to None. A valid python typing type (e.g. Optional[List[int]]) for the value contained within the DagsterType. Meant for internal use.
</dd>
</dl>
</div>
---
---
title: 'dagster-pandera library'
sidebar_position: 1000
title_meta: 'dagster-pandera library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-pandera library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pandera-library">
# dagster-pandera library
The <cite>dagster_pandera</cite> library allows Dagster users to use dataframe validation library [Pandera](https://github.com/pandera-dev/pandera) for the validation of Pandas dataframes. See [the guide](https://docs.dagster.io/integrations/libraries/pandera) for details.
<dl>
<dt><Link class="anchor" id='dagster_pandera.pandera_schema_to_dagster_type'>dagster_pandera.pandera_schema_to_dagster_type<a href="#dagster_pandera.pandera_schema_to_dagster_type" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Convert a Pandera dataframe schema to a <cite>DagsterType</cite>.
The generated Dagster type will be given an automatically generated <cite>name</cite>. The schema’s <cite>title</cite>
property, <cite>name</cite> property, or class name (in that order) will be used. If neither <cite>title</cite> or
<cite>name</cite> is defined, a name of the form <cite>DagsterPanderaDataframe\<n></cite> is generated.
Additional metadata is also extracted from the Pandera schema and attached to the returned
<cite>DagsterType</cite> as a metadata dictionary. The extracted metadata includes:
- Descriptions on the schema and constituent columns and checks.
- Data types for each column.
- String representations of all column-wise checks.
- String representations of all row-wise (i.e. “wide”) checks.
The returned <cite>DagsterType</cite> type will call the Pandera schema’s <cite>validate()</cite> method in its type
check function. Validation is done in <cite>lazy</cite> mode, i.e. pandera will attempt to validate all
values in the dataframe, rather than stopping on the first error.
If validation fails, the returned <cite>TypeCheck</cite> object will contain two pieces of metadata:
- <cite>num_failures</cite> total number of validation errors.
- <cite>failure_sample</cite> a table containing up to the first 10 validation errors.
Parameters: <strong>schema</strong> (<em>Union</em><em>[</em><em>pa.DataFrameSchema</em><em>, </em><em>Type</em><em>[</em><em>pa.DataFrameModel</em><em>]</em><em>]</em>)Returns: Dagster Type constructed from the Pandera schema.Return type: [DagsterType](../../../api/dagster/types.mdx#dagster.DagsterType)
</dd>
</dl>
</div>
---
---
title: 'dagster-papertrail'
sidebar_position: 1000
title_meta: 'dagster-papertrail API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-papertrail Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-papertrail">
# dagster-papertrail
This library provides an integration with [Papertrail](https://papertrailapp.com) for logging.
You can easily set up your Dagster job to log to Papertrail. You’ll need an active Papertrail
account, and have your papertrail URL and port handy.
<dl>
<dt><Link class="anchor" id='dagster_papertrail.papertrail_logger'>dagster_papertrail.papertrail_logger LoggerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/definitions/logger_definition.py#L50' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_papertrail.papertrail_logger" class="hash-link"></a></Link></dt>
<dd>
Core class for defining loggers.
Loggers are job-scoped logging handlers, which will be automatically invoked whenever
dagster messages are logged from within a job.
Parameters:
- <strong>logger_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*InitLoggerContext*](../../../api/dagster/loggers.mdx#dagster.InitLoggerContext)<em>]</em><em>, </em><em>logging.Logger</em><em>]</em>) – User-provided function to instantiate the logger. This logger will be automatically invoked whenever the methods on `context.log` are called from within job compute logic.
- <strong>config_schema</strong> (<em>Optional</em><em>[</em>[*ConfigSchema*](../../../api/dagster/config.mdx#dagster.ConfigSchema)<em>]</em>) – The schema for the config. Configuration data available in <cite>init_context.logger_config</cite>. If not set, Dagster will accept any config provided.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A human-readable description of this logger.
</dd>
</dl>
</div>
---
---
title: 'dagster-pipes library'
sidebar_position: 1000
title_meta: 'dagster-pipes library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-pipes library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pipes-library">
# dagster-pipes library
The `dagster-pipes` library is intended for inclusion in an external process that integrates with Dagster using the [Pipes](https://docs.dagster.io/integrations/external-pipelines) protocol. This could be in an environment like Databricks, Kubernetes, or Docker. Using this library, you can write code in the external process that streams metadata back to Dagster.
For a detailed look at the Pipes process, including how to customize it, see [Dagster Pipes details and customization](https://docs.dagster.io/integrations/external-pipelines/dagster-pipes-details-and-customization).
<strong>Looking to set up a Pipes client in Dagster?</strong> See the [Dagster Pipes API reference](https://docs.dagster.io/api/dagster/pipe).
<strong>Note</strong>: This library isn’t included with `dagster` and must be [installed separately](https://pypi.org/project/dagster-pipes).
<div class="section" id="context">
## Context
<dl>
<dt><Link class="anchor" id='dagster_pipes.open_dagster_pipes'>dagster_pipes.open_dagster_pipes <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1534' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.open_dagster_pipes" class="hash-link"></a></Link></dt>
<dd>
Initialize the Dagster Pipes context.
This function should be called near the entry point of a pipes process. It will load injected
context information from Dagster and spin up the machinery for streaming messages back to
Dagster.
If the process was not launched by Dagster, this function will emit a warning and return a
<cite>MagicMock</cite> object. This should make all operations on the context no-ops and prevent your code
from crashing.
Parameters:
- <strong>context_loader</strong> (<em>Optional</em><em>[</em>[*PipesContextLoader*](#dagster_pipes.PipesContextLoader)<em>]</em>) – The context loader to use. Defaults to [`PipesDefaultContextLoader`](#dagster_pipes.PipesDefaultContextLoader).
- <strong>message_writer</strong> (<em>Optional</em><em>[</em>[*PipesMessageWriter*](#dagster_pipes.PipesMessageWriter)<em>]</em>) – The message writer to use. Defaults to [`PipesDefaultMessageWriter`](#dagster_pipes.PipesDefaultMessageWriter).
- <strong>params_loader</strong> (<em>Optional</em><em>[</em>[*PipesParamsLoader*](#dagster_pipes.PipesParamsLoader)<em>]</em>) – The params loader to use. Defaults to [`PipesEnvVarParamsLoader`](#dagster_pipes.PipesEnvVarParamsLoader).
Returns: The initialized context.Return type: [PipesContext](#dagster_pipes.PipesContext)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext'>`class` dagster_pipes.PipesContext <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1576' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext" class="hash-link"></a></Link></dt>
<dd>
The context for a Dagster Pipes process.
This class is analogous to [`OpExecutionContext`](../../../api/dagster/execution.mdx#dagster.OpExecutionContext) on the Dagster side of the Pipes
connection. It provides access to information such as the asset key(s) and partition key(s) in
scope for the current step. It also provides methods for logging and emitting results that will
be streamed back to Dagster.
This class should not be directly instantiated by the user. Instead it should be initialized by
calling [`open_dagster_pipes()`](#dagster_pipes.open_dagster_pipes), which will return the singleton instance of this class.
After <cite>open_dagster_pipes()</cite> has been called, the singleton instance can also be retrieved by
calling [`PipesContext.get()`](#dagster_pipes.PipesContext.get).
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.get'>`classmethod` get <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1602' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.get" class="hash-link"></a></Link></dt>
<dd>
Get the singleton instance of the context. Raises an error if the context has not been initialized.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.is_initialized'>`classmethod` is_initialized <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1592' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.is_initialized" class="hash-link"></a></Link></dt>
<dd>
bool: Whether the context has been initialized.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.set'>`classmethod` set <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1597' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.set" class="hash-link"></a></Link></dt>
<dd>
Set the singleton instance of the context.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.close'>close <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1640' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.close" class="hash-link"></a></Link></dt>
<dd>
Close the pipes connection. This will flush all buffered messages to the orchestration
process and cause any further attempt to write a message to raise an error. This method is
idempotent– subsequent calls after the first have no effect.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.get_extra'>get_extra <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1788' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.get_extra" class="hash-link"></a></Link></dt>
<dd>
Get the value of an extra provided by the user. Raises an error if the extra is not defined.
Parameters: <strong>key</strong> (<em>str</em>) – The key of the extra.Returns: The value of the extra.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.log_external_stream'>log_external_stream <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1898' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.log_external_stream" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.report_asset_check'>report_asset_check <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1847' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.report_asset_check" class="hash-link"></a></Link></dt>
<dd>
Report to Dagster that an asset check has been performed. Streams a payload containing
check result information back to Dagster. If no assets or associated checks are in scope, raises an error.
Parameters:
- <strong>check_name</strong> (<em>str</em>) – The name of the check.
- <strong>passed</strong> (<em>bool</em>) – Whether the check passed.
- <strong>severity</strong> (<em>PipesAssetCheckSeverity</em>) – The severity of the check. Defaults to “ERROR”.
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>PipesMetadataRawValue</em><em>, </em><em>PipesMetadataValue</em><em>]</em><em>]</em><em>]</em>) – Metadata for the check. Defaults to None.
- <strong>asset_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The asset key for the check. If only a single asset is in scope, default to that asset’s key. If multiple assets are in scope, this must be set explicitly or an error will be raised.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.report_asset_materialization'>report_asset_materialization <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1806' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.report_asset_materialization" class="hash-link"></a></Link></dt>
<dd>
Report to Dagster that an asset has been materialized. Streams a payload containing
materialization information back to Dagster. If no assets are in scope, raises an error.
Parameters:
- <strong>metadata</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Union</em><em>[</em><em>PipesMetadataRawValue</em><em>, </em><em>PipesMetadataValue</em><em>]</em><em>]</em><em>]</em>) – Metadata for the materialized asset. Defaults to None.
- <strong>data_version</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The data version for the materialized asset. Defaults to None.
- <strong>asset_key</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The asset key for the materialized asset. If only a single asset is in scope, default to that asset’s key. If multiple assets are in scope, this must be set explicitly or an error will be raised.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.report_custom_message'>report_custom_message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1889' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.report_custom_message" class="hash-link"></a></Link></dt>
<dd>
Send a JSON serializable payload back to the orchestration process. Can be retrieved there
using <cite>get_custom_messages</cite>.
Parameters: <strong>payload</strong> (<em>Any</em>) – JSON serializable data.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.asset_key'>`property` asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1674' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.asset_key" class="hash-link"></a></Link></dt>
<dd>
The AssetKey for the currently scoped asset. Raises an error if 0 or multiple assets
are in scope.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.asset_keys'>`property` asset_keys <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1683' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.asset_keys" class="hash-link"></a></Link></dt>
<dd>
The AssetKeys for the currently scoped assets. Raises an error if no
assets are in scope.
Type: Sequence[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.code_version'>`property` code_version <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1712' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.code_version" class="hash-link"></a></Link></dt>
<dd>
The code version for the currently scoped asset. Raises an error if 0 or
multiple assets are in scope.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.code_version_by_asset_key'>`property` code_version_by_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1723' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.code_version_by_asset_key" class="hash-link"></a></Link></dt>
<dd>
Mapping of asset key to code version for the currently
scoped assets. Raises an error if no assets are in scope.
Type: Mapping[str, Optional[str]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.extras'>`property` extras <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1799' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.extras" class="hash-link"></a></Link></dt>
<dd>
Key-value map for all extras provided by the user.
Type: Mapping[str, Any]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.is_asset_step'>`property` is_asset_step <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1669' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.is_asset_step" class="hash-link"></a></Link></dt>
<dd>
Whether the current step targets assets.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.is_closed'>`property` is_closed <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1654' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.is_closed" class="hash-link"></a></Link></dt>
<dd>
Whether the context has been closed.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.is_partition_step'>`property` is_partition_step <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1733' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.is_partition_step" class="hash-link"></a></Link></dt>
<dd>
Whether the current step is scoped to one or more partitions.
Type: bool
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.job_name'>`property` job_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1776' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.job_name" class="hash-link"></a></Link></dt>
<dd>
The job name for the currently executing run. Returns None if the run is
not derived from a job.
Type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.log'>`property` log <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1903' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.log" class="hash-link"></a></Link></dt>
<dd>
A logger that streams log messages back to Dagster.
Type: logging.Logger
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.partition_key'>`property` partition_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1738' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.partition_key" class="hash-link"></a></Link></dt>
<dd>
The partition key for the currently scoped partition. Raises an error if 0 or
multiple partitions are in scope.
Type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.partition_key_range'>`property` partition_key_range <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1748' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.partition_key_range" class="hash-link"></a></Link></dt>
<dd>
The partition key range for the currently scoped partition or
partitions. Raises an error if no partitions are in scope.
Type: PipesPartitionKeyRange
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.partition_time_window'>`property` partition_time_window <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1758' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.partition_time_window" class="hash-link"></a></Link></dt>
<dd>
The partition time window for the currently scoped partition
or partitions. Returns None if partitions in scope are not temporal. Raises an error if no
partitions are in scope.
Type: Optional[PipesTimeWindow]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.provenance'>`property` provenance <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1691' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.provenance" class="hash-link"></a></Link></dt>
<dd>
The provenance for the currently scoped asset. Raises an
error if 0 or multiple assets are in scope.
Type: Optional[PipesDataProvenance]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.provenance_by_asset_key'>`property` provenance_by_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1702' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.provenance_by_asset_key" class="hash-link"></a></Link></dt>
<dd>
Mapping of asset key to provenance for the
currently scoped assets. Raises an error if no assets are in scope.
Type: Mapping[str, Optional[PipesDataProvenance]]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.retry_number'>`property` retry_number <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1783' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.retry_number" class="hash-link"></a></Link></dt>
<dd>
The retry number for the currently executing run.
Type: int
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContext.run_id'>`property` run_id <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1771' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContext.run_id" class="hash-link"></a></Link></dt>
<dd>
The run ID for the currently executing pipeline run.
Type: str
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="advanced">
## Advanced
Most Pipes users won’t need to use the APIs in the following sections unless they are customizing the Pipes protocol.
Refer to the [Dagster Pipes details and customization guide](https://docs.dagster.io/integrations/external-pipelines/dagster-pipes-details-and-customization) for more information.
<div class="section" id="context-loaders">
### Context loaders
Context loaders load the context payload from the location specified in the bootstrap payload.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContextLoader'>`class` dagster_pipes.PipesContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L497' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContextLoader" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesContextLoader.load_context'>`abstractmethod` load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L498' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultContextLoader'>`class` dagster_pipes.PipesDefaultContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L734' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDefaultContextLoader" class="hash-link"></a></Link></dt>
<dd>
Context loader that loads context data from either a file or directly from the provided params.
The location of the context data is configured by the params received by the loader. If the params
include a key <cite>path</cite>, then the context data will be loaded from a file at the specified path. If
the params instead include a key <cite>data</cite>, then the corresponding value should be a dict
representing the context data.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultContextLoader.load_context'>load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L746' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDefaultContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultContextLoader.DIRECT_KEY'>DIRECT_KEY `=` 'data'<a href="#dagster_pipes.PipesDefaultContextLoader.DIRECT_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultContextLoader.FILE_PATH_KEY'>FILE_PATH_KEY `=` 'path'<a href="#dagster_pipes.PipesDefaultContextLoader.FILE_PATH_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3ContextLoader'>`class` dagster_pipes.PipesS3ContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1184' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3ContextLoader" class="hash-link"></a></Link></dt>
<dd>
Context loader that reads context from a JSON file on S3.
Parameters: <strong>client</strong> (<em>Any</em>) – A boto3.client(“s3”) object.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3ContextLoader.load_context'>load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1194' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3ContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSContextLoader'>`class` dagster_pipes.PipesGCSContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1262' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSContextLoader" class="hash-link"></a></Link></dt>
<dd>
Context loader that reads context from a JSON file on GCS.
Parameters: <strong>client</strong> (<em>google.cloud.storage.Client</em>) – A google.cloud.storage.Client object.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSContextLoader.load_context'>load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1272' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDbfsContextLoader'>`class` dagster_pipes.PipesDbfsContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1410' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDbfsContextLoader" class="hash-link"></a></Link></dt>
<dd>
Context loader that reads context from a JSON file on DBFS.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDbfsContextLoader.load_context'>load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1413' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDbfsContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageContextLoader'>`class` dagster_pipes.PipesAzureBlobStorageContextLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1336' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageContextLoader" class="hash-link"></a></Link></dt>
<dd>
Context loader that reads context from a JSON file on AzureBlobStorage.
Parameters: <strong>client</strong> (<em>Any</em>) – An azure.storage.blob.BlobServiceClient object.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageContextLoader.load_context'>load_context <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1346' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageContextLoader.load_context" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that loads context data injected by the orchestration process.
This method should read and yield the context data from the location specified by the passed in
<cite>PipesParams</cite>.
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the context injector in the orchestration
process.Yields: <em>PipesContextData</em> – The context data.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="params-loaders">
### Params loaders
Params loaders load the bootstrap payload from some globally accessible key-value store.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesParamsLoader'>`class` dagster_pipes.PipesParamsLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L597' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesParamsLoader" class="hash-link"></a></Link></dt>
<dd>
Object that loads params passed from the orchestration process by the context injector and
message reader. These params are used to respectively bootstrap the
[`PipesContextLoader`](#dagster_pipes.PipesContextLoader) and [`PipesMessageWriter`](#dagster_pipes.PipesMessageWriter).
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesParamsLoader.is_dagster_pipes_process'>`abstractmethod` is_dagster_pipes_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L603' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesParamsLoader.is_dagster_pipes_process" class="hash-link"></a></Link></dt>
<dd>
Whether or not this process has been provided with provided with information to create
a PipesContext or should instead return a mock.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesParamsLoader.load_context_params'>`abstractmethod` load_context_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L609' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesParamsLoader.load_context_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side context injector.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesParamsLoader.load_messages_params'>`abstractmethod` load_messages_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L613' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesParamsLoader.load_messages_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side message reader.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesEnvVarParamsLoader'>`class` dagster_pipes.PipesEnvVarParamsLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1091' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesEnvVarParamsLoader" class="hash-link"></a></Link></dt>
<dd>
Params loader that extracts params from environment variables.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesCliArgsParamsLoader'>`class` dagster_pipes.PipesCliArgsParamsLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1118' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesCliArgsParamsLoader" class="hash-link"></a></Link></dt>
<dd>
Params loader that extracts params from known CLI arguments.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesCliArgsParamsLoader.is_dagster_pipes_process'>is_dagster_pipes_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1124' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesCliArgsParamsLoader.is_dagster_pipes_process" class="hash-link"></a></Link></dt>
<dd>
Whether or not this process has been provided with provided with information to create
a PipesContext or should instead return a mock.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesCliArgsParamsLoader.load_context_params'>load_context_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1128' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesCliArgsParamsLoader.load_context_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side context injector.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesCliArgsParamsLoader.load_messages_params'>load_messages_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1132' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesCliArgsParamsLoader.load_messages_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side message reader.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMappingParamsLoader'>`class` dagster_pipes.PipesMappingParamsLoader <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1072' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMappingParamsLoader" class="hash-link"></a></Link></dt>
<dd>
Params loader that extracts params from a Mapping provided at init time.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMappingParamsLoader.is_dagster_pipes_process'>is_dagster_pipes_process <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1078' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMappingParamsLoader.is_dagster_pipes_process" class="hash-link"></a></Link></dt>
<dd>
Whether or not this process has been provided with provided with information to create
a PipesContext or should instead return a mock.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMappingParamsLoader.load_context_params'>load_context_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1082' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMappingParamsLoader.load_context_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side context injector.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMappingParamsLoader.load_messages_params'>load_messages_params <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1086' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMappingParamsLoader.load_messages_params" class="hash-link"></a></Link></dt>
<dd>
PipesParams: Load params passed by the orchestration-side message reader.
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="message-writers">
### Message writers
Message writers write messages to the location specified in the bootstrap payload.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriter'>`class` dagster_pipes.PipesMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L546' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriter" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriter.get_opened_extras'>get_opened_extras <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L575' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriter.get_opened_extras" class="hash-link"></a></Link></dt>
<dd>
Return arbitary reader-specific information to be passed back to the orchestration
process under the <cite>extras</cite> key of the initialization payload.
Returns: A dict of arbitrary data to be passed back to the orchestration process.Return type: PipesExtras
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriter.get_opened_payload'>`final` get_opened_payload <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L564' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriter.get_opened_payload" class="hash-link"></a></Link></dt>
<dd>
Return a payload containing information about the external process to be passed back to
the orchestration process. This should contain information that cannot be known before
the external process is launched.
This method should not be overridden by users. Instead, users should
override <cite>get_opened_extras</cite> to inject custom data.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriter.open'>`abstractmethod` open <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L547' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriter.open" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that initializes a channel for writing messages back to Dagster.
This method should takes the params passed by the orchestration-side
`PipesMessageReader` and use them to construct and yield a
[`PipesMessageWriterChannel`](#dagster_pipes.PipesMessageWriterChannel).
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the message reader in the orchestration
process.Yields: <em>PipesMessageWriterChannel</em> – Channel for writing messagse back to Dagster.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter'>`class` dagster_pipes.PipesDefaultMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L916' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDefaultMessageWriter" class="hash-link"></a></Link></dt>
<dd>
Message writer that writes messages to either a file or the stdout or stderr stream.
The write location is configured by the params received by the writer. If the params include a
key <cite>path</cite>, then messages will be written to a file at the specified path. If the params instead
include a key <cite>stdio</cite>, then messages then the corresponding value must specify either <cite>stderr</cite>
or <cite>stdout</cite>, and messages will be written to the selected stream.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.open'>open <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L932' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDefaultMessageWriter.open" class="hash-link"></a></Link></dt>
<dd>
A <cite>@contextmanager</cite> that initializes a channel for writing messages back to Dagster.
This method should takes the params passed by the orchestration-side
`PipesMessageReader` and use them to construct and yield a
[`PipesMessageWriterChannel`](#dagster_pipes.PipesMessageWriterChannel).
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the message reader in the orchestration
process.Yields: <em>PipesMessageWriterChannel</em> – Channel for writing messagse back to Dagster.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.BUFFERED_STDIO_KEY'>BUFFERED_STDIO_KEY `=` 'buffered_stdio'<a href="#dagster_pipes.PipesDefaultMessageWriter.BUFFERED_STDIO_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.FILE_PATH_KEY'>FILE_PATH_KEY `=` 'path'<a href="#dagster_pipes.PipesDefaultMessageWriter.FILE_PATH_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.INCLUDE_STDIO_IN_MESSAGES_KEY'>INCLUDE_STDIO_IN_MESSAGES_KEY `=` 'include_stdio_in_messages'<a href="#dagster_pipes.PipesDefaultMessageWriter.INCLUDE_STDIO_IN_MESSAGES_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.STDERR'>STDERR `=` 'stderr'<a href="#dagster_pipes.PipesDefaultMessageWriter.STDERR" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.STDIO_KEY'>STDIO_KEY `=` 'stdio'<a href="#dagster_pipes.PipesDefaultMessageWriter.STDIO_KEY" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDefaultMessageWriter.STDOUT'>STDOUT `=` 'stdout'<a href="#dagster_pipes.PipesDefaultMessageWriter.STDOUT" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriter'>`class` dagster_pipes.PipesBlobStoreMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L623' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriter" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriter.make_channel'>`abstractmethod` make_channel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L658' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriter.make_channel" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriter.open'>open <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L632' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriter.open" class="hash-link"></a></Link></dt>
<dd>
Construct and yield a [`PipesBlobStoreMessageWriterChannel`](#dagster_pipes.PipesBlobStoreMessageWriterChannel).
Parameters: <strong>params</strong> (<em>PipesParams</em>) – The params provided by the message reader in the orchestration
process.Yields: <em>PipesBlobStoreMessageWriterChannel</em> – Channel that periodically uploads message chunks to
a blob store.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriter.INCLUDE_STDIO_IN_MESSAGES_KEY'>INCLUDE_STDIO_IN_MESSAGES_KEY `=` 'include_stdio_in_messages'<a href="#dagster_pipes.PipesBlobStoreMessageWriter.INCLUDE_STDIO_IN_MESSAGES_KEY" class="hash-link"></a></Link></dt>
<dd>
Message writer channel that periodically uploads message chunks to some blob store endpoint.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3MessageWriter'>`class` dagster_pipes.PipesS3MessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1202' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3MessageWriter" class="hash-link"></a></Link></dt>
<dd>
Message writer that writes messages by periodically writing message chunks to an S3 bucket.
Parameters:
- <strong>client</strong> (<em>Any</em>) – A boto3.client(“s3”) object.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3MessageWriter.make_channel'>make_channel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1217' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3MessageWriter.make_channel" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSMessageWriter'>`class` dagster_pipes.PipesGCSMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1280' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSMessageWriter" class="hash-link"></a></Link></dt>
<dd>
Message writer that writes messages by periodically writing message chunks to a GCS bucket.
Parameters:
- <strong>client</strong> (<em>google.cloud.storage.Client</em>) – A google.cloud.storage.Client object.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSMessageWriter.make_channel'>make_channel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1292' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSMessageWriter.make_channel" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDbfsMessageWriter'>`class` dagster_pipes.PipesDbfsMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1421' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDbfsMessageWriter" class="hash-link"></a></Link></dt>
<dd>
Message writer that writes messages by periodically writing message chunks to a directory on DBFS.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDbfsMessageWriter.get_opened_extras'>get_opened_extras <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1434' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDbfsMessageWriter.get_opened_extras" class="hash-link"></a></Link></dt>
<dd>
Return arbitary reader-specific information to be passed back to the orchestration
process under the <cite>extras</cite> key of the initialization payload.
Returns: A dict of arbitrary data to be passed back to the orchestration process.Return type: PipesExtras
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesDbfsMessageWriter.make_channel'>make_channel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1424' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesDbfsMessageWriter.make_channel" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageMessageWriter'>`class` dagster_pipes.PipesAzureBlobStorageMessageWriter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1355' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageMessageWriter" class="hash-link"></a></Link></dt>
<dd>
Message writer that writes messages by periodically writing message chunks to an
AzureBlobStorage container.
Parameters:
- <strong>client</strong> (<em>Any</em>) – An azure.storage.blob.BlobServiceClient object.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageMessageWriter.make_channel'>make_channel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1368' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageMessageWriter.make_channel" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="message-writer-channels">
### Message writer channels
Message writer channels are objects that write messages back to the Dagster orchestration process.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriterChannel'>`class` dagster_pipes.PipesMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L585' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Object that writes messages back to the Dagster orchestration process.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesMessageWriterChannel.write_message'>`abstractmethod` write_message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L588' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesMessageWriterChannel.write_message" class="hash-link"></a></Link></dt>
<dd>
Write a message to the orchestration process.
Parameters: <strong>message</strong> (<em>PipesMessage</em>) – The message to write.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriterChannel'>`class` dagster_pipes.PipesBlobStoreMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L662' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel that periodically uploads message chunks to some blob store endpoint.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriterChannel.buffered_upload_loop'>buffered_upload_loop <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L682' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriterChannel.buffered_upload_loop" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriterChannel.flush_messages'>flush_messages <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L673' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriterChannel.flush_messages" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriterChannel.upload_messages_chunk'>`abstractmethod` upload_messages_chunk <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L679' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriterChannel.upload_messages_chunk" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBlobStoreMessageWriterChannel.write_message'>write_message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L670' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBlobStoreMessageWriterChannel.write_message" class="hash-link"></a></Link></dt>
<dd>
Write a message to the orchestration process.
Parameters: <strong>message</strong> (<em>PipesMessage</em>) – The message to write.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBufferedFilesystemMessageWriterChannel'>`class` dagster_pipes.PipesBufferedFilesystemMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L712' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBufferedFilesystemMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel that periodically writes message chunks to an endpoint mounted on the filesystem.
Parameters: <strong>interval</strong> (<em>float</em>) – interval in seconds between chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesBufferedFilesystemMessageWriterChannel.upload_messages_chunk'>upload_messages_chunk <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L723' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesBufferedFilesystemMessageWriterChannel.upload_messages_chunk" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesFileMessageWriterChannel'>`class` dagster_pipes.PipesFileMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L981' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesFileMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel that writes one message per line to a file.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesFileMessageWriterChannel.write_message'>write_message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L987' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesFileMessageWriterChannel.write_message" class="hash-link"></a></Link></dt>
<dd>
Write a message to the orchestration process.
Parameters: <strong>message</strong> (<em>PipesMessage</em>) – The message to write.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesStreamMessageWriterChannel'>`class` dagster_pipes.PipesStreamMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L992' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesStreamMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel that writes one message per line to a <cite>TextIO</cite> stream.
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesStreamMessageWriterChannel.write_message'>write_message <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L998' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesStreamMessageWriterChannel.write_message" class="hash-link"></a></Link></dt>
<dd>
Write a message to the orchestration process.
Parameters: <strong>message</strong> (<em>PipesMessage</em>) – The message to write.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3MessageWriterChannel'>`class` dagster_pipes.PipesS3MessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1231' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3MessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel for writing messages by periodically writing message chunks to an S3 bucket.
Parameters:
- <strong>client</strong> (<em>Any</em>) – A boto3.client(“s3”) object.
- <strong>bucket</strong> (<em>str</em>) – The name of the S3 bucket to write to.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the keys of written blobs.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesS3MessageWriterChannel.upload_messages_chunk'>upload_messages_chunk <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1248' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesS3MessageWriterChannel.upload_messages_chunk" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSMessageWriterChannel'>`class` dagster_pipes.PipesGCSMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1306' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel for writing messages by periodically writing message chunks to a GCS bucket.
Parameters:
- <strong>client</strong> (<em>google.cloud.storage.Client</em>) – A google.cloud.storage.Client object.
- <strong>bucket</strong> (<em>str</em>) – The name of the GCS bucket to write to.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the keys of written blobs.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesGCSMessageWriterChannel.upload_messages_chunk'>upload_messages_chunk <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1326' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesGCSMessageWriterChannel.upload_messages_chunk" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageMessageWriterChannel'>`class` dagster_pipes.PipesAzureBlobStorageMessageWriterChannel <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1382' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageMessageWriterChannel" class="hash-link"></a></Link></dt>
<dd>
Message writer channel for writing messages by periodically writing message chunks to an
AzureBlobStorage container.
Parameters:
- <strong>client</strong> (<em>Any</em>) – An azure.storage.blob.BlobServiceClient object.
- <strong>bucket</strong> (<em>str</em>) – The name of the AzureBlobStorage container to write to.
- <strong>key_prefix</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – An optional prefix to use for the keys of written blobs.
- <strong>interval</strong> (<em>float</em>) – interval in seconds between upload chunk uploads
<dl>
<dt><Link class="anchor" id='dagster_pipes.PipesAzureBlobStorageMessageWriterChannel.upload_messages_chunk'>upload_messages_chunk <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L1399' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.PipesAzureBlobStorageMessageWriterChannel.upload_messages_chunk" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="utilities">
### Utilities
<dl>
<dt><Link class="anchor" id='dagster_pipes.encode_env_var'>dagster_pipes.encode_env_var <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L409' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.encode_env_var" class="hash-link"></a></Link></dt>
<dd>
Encode value by serializing to JSON, compressing with zlib, and finally encoding with base64.
<cite>base64_encode(compress(to_json(value)))</cite> in function notation.
Parameters: <strong>value</strong> (<em>Any</em>) – The value to encode. Must be JSON-serializable.Returns: The encoded value.Return type: str
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.decode_env_var'>dagster_pipes.decode_env_var <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L425' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.decode_env_var" class="hash-link"></a></Link></dt>
<dd>
Decode a value by decoding from base64, decompressing with zlib, and finally deserializing from
JSON. <cite>from_json(decompress(base64_decode(value)))</cite> in function notation.
Parameters: <strong>value</strong> (<em>Any</em>) – The value to decode.Returns: The decoded value.Return type: Any
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.DagsterPipesError'>`class` dagster_pipes.DagsterPipesError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L228' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.DagsterPipesError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_pipes.DagsterPipesWarning'>`class` dagster_pipes.DagsterPipesWarning <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py#L232' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pipes.DagsterPipesWarning" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-polars library'
sidebar_position: 1000
title_meta: 'dagster-polars library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-polars library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-polars-library">
# dagster-polars library
This library provides Dagster integration with [Polars](https://pola.rs).
It allows using Polars eager or lazy DataFrames as inputs and outputs with Dagster’s <cite>@asset</cite> and <cite>@op</cite>.
Type annotations are used to control whether to load an eager or lazy DataFrame. Lazy DataFrames can be sinked as output.
Multiple serialization formats (Parquet, Delta Lake, BigQuery) and filesystems (local, S3, GCS, …) are supported.
A comprehensive list of <cite>dagster-polars</cite> behavior for supported type annotations can be found in [Type annotations](#types)`Type annotations` section.
<strong>Note:</strong> This is a community-supported integration. For support, see the [Dagster Community Integrations repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-polars).
<div class="section" id="installation">
## Installation
```default
pip install dagster-polars
```
Some IOManagers (like [`PolarsDeltaIOManager`](#dagster_polars.PolarsDeltaIOManager)) may require additional dependencies, which are provided with extras like <cite>dagster-polars[delta]</cite>.
Please check the documentation for each IOManager for more details.
</div>
<div class="section" id="quickstart">
## Quickstart
Common filesystem-based IOManagers features highlights, using [`PolarsParquetIOManager`](#dagster_polars.PolarsParquetIOManager) as an example (see [`BasePolarsUPathIOManager`](#dagster_polars.BasePolarsUPathIOManager) for the full list of features provided by <cite>dagster-polars</cite>):
Type annotations are not required. By default an eager <cite>pl.DataFrame</cite> will be loaded.
```python
from dagster import asset
import polars as pl
@asset(io_manager_key="polars_parquet_io_manager")
def upstream():
return DataFrame({"foo": [1, 2, 3]})
@asset(io_manager_key="polars_parquet_io_manager")
def downstream(upstream) -> pl.LazyFrame:
assert isinstance(upstream, pl.DataFrame)
return upstream.lazy() # LazyFrame will be sinked
```
Lazy <cite>pl.LazyFrame</cite> can be scanned by annotating the input with <cite>pl.LazyFrame</cite>, and returning a <cite>pl.LazyFrame</cite> will sink it:
```python
@asset(io_manager_key="polars_parquet_io_manager")
def downstream(upstream: pl.LazyFrame) -> pl.LazyFrame:
assert isinstance(upstream, pl.LazyFrame)
return upstream
```
The same logic applies to partitioned assets:
```python
@asset
def downstream(partitioned_upstream: Dict[str, pl.LazyFrame]):
assert isinstance(partitioned_upstream, dict)
assert isinstance(partitioned_upstream["my_partition"], pl.LazyFrame)
```
<cite>Optional</cite> inputs and outputs are supported:
```python
@asset
def upstream() -> Optional[pl.DataFrame]:
if has_data:
return DataFrame({"foo": [1, 2, 3]}) # type check will pass
else:
return None # type check will pass and `dagster_polars` will skip writing the output completely
@asset
def downstream(upstream: Optional[pl.LazyFrame]): # upstream will be None if it doesn't exist in storage
...
```
By default all the IOManagers store separate partitions as physically separated locations, such as:
- <cite>/my/asset/key/partition_0.extension</cite>
- <cite>/my/asset/key/partition_1.extension</cite>
This mode is useful for e.g. snapshotting.
Some IOManagers (like [`PolarsDeltaIOManager`](#dagster_polars.PolarsDeltaIOManager)) support reading and writing partitions in storage-native format in the same location.
This mode can be typically enabled by setting <cite>“partition_by”</cite> metadata value. For example, [`PolarsDeltaIOManager`](#dagster_polars.PolarsDeltaIOManager) would store different partitions in the same <cite>/my/asset/key.delta</cite> directory, which will be properly partitioned.
This mode should be preferred for true partitioning.
</div>
<div class="section" id="type-annotations">
<Link id="types"></Link>
## Type annotations
Type aliases like <cite>DataFrameWithPartitions</cite> are provided by `dagster_polars.types` for convenience.
## Supported type annotations and <cite>dagster-polars</cite> behavior
| Type annotation | Type Alias | Behavior |
| :------------------------ | :--------- | :-------------------------------------------------------------------------- |
| <cite>DataFrame</cite> | | read/write a<cite>DataFrame</cite> |
| <cite>LazyFrame</cite> | | read/sink a<cite>LazyFrame</cite> |
| <cite>Optional[DataFrame]</cite> | | read/write a<cite>DataFrame</cite>. Do nothing if no data is found in storage or the output is<cite>None</cite> |
| <cite>Optional[LazyFrame]</cite> | | read a<cite>LazyFrame</cite>. Do nothing if no data is found in storage |
| <cite>Dict[str, DataFrame]</cite> | <cite>DataFrameWithPartitions</cite> | read multiple<cite>DataFrame`s as `Dict[str, DataFrame]</cite>. Raises an error for missing partitions, unless<cite>“allow_missing_partitions”</cite>input metadata is set to<cite>True</cite> |
| <cite>Dict[str, LazyFrame]</cite> | <cite>LazyFramePartitions</cite> | read multiple<cite>LazyFrame`s as `Dict[str, LazyFrame]</cite>. Raises an error for missing partitions, unless<cite>“allow_missing_partitions”</cite>input metadata is set to<cite>True</cite> |
Generic builtins (like <cite>tuple[…]</cite> instead of <cite>Tuple[…]</cite>) are supported for Python >= 3.9.
</div>
<div class="section" id="api-documentation">
## API documentation
<dl>
<dt><Link class="anchor" id='dagster_polars.BasePolarsUPathIOManager'>dagster_polars.BasePolarsUPathIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_polars/io_managers/base.py#L42' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_polars.BasePolarsUPathIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for <cite>dagster-polars</cite> IOManagers.
Doesn’t define a specific storage format.
To implement a specific storage format (parquet, csv, etc), inherit from this class and implement the <cite>write_df_to_path</cite>, <cite>sink_df_to_path</cite> and <cite>scan_df_from_path</cite> methods.
Features:
- All the features of [`UPathIOManager`](../../../api/dagster/io-managers.mdx#dagster.UPathIOManager) - works with local and remote filesystems (like S3), supports loading multiple partitions with respect to [`PartitionMapping`](../../../api/dagster/partitions.mdx#dagster.PartitionMapping), and more
- loads the correct type - <cite>polars.DataFrame</cite>, <cite>polars.LazyFrame</cite>, or other types defined in `dagster_polars.types` - based on the input type annotation (or <cite>dagster.DagsterType</cite>’s <cite>typing_type</cite>)
- can sink lazy <cite>pl.LazyFrame</cite> DataFrames
- handles <cite>Nones</cite> with <cite>Optional</cite> types by skipping loading missing inputs or saving <cite>None</cite> outputs
- logs various metadata about the DataFrame - size, schema, sample, stats, …
- the <cite>“columns”</cite> input metadata value can be used to select a subset of columns to load
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_polars.PolarsParquetIOManager'>dagster_polars.PolarsParquetIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_polars/io_managers/parquet.py#L84' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_polars.PolarsParquetIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Implements reading and writing Polars DataFrames in Apache Parquet format.
Features:
- All features provided by [`BasePolarsUPathIOManager`](#dagster_polars.BasePolarsUPathIOManager).
- All read/write options can be set via corresponding metadata or config parameters (metadata takes precedence).
- Supports reading partitioned Parquet datasets (for example, often produced by Spark).
- Supports reading/writing custom metadata in the Parquet file’s schema as json-serialized bytes at <cite>“dagster_polars_metadata”</cite> key.
Examples:
```python
from dagster import asset
from dagster_polars import PolarsParquetIOManager
import polars as pl
@asset(
io_manager_key="polars_parquet_io_manager",
key_prefix=["my_dataset"]
)
def my_asset() -> pl.DataFrame: # data will be stored at <base_dir>/my_dataset/my_asset.parquet
...
defs = Definitions(
assets=[my_table],
resources={
"polars_parquet_io_manager": PolarsParquetIOManager(base_dir="s3://my-bucket/my-dir")
}
)
```
Reading partitioned Parquet datasets:
```python
from dagster import SourceAsset
my_asset = SourceAsset(
key=["path", "to", "dataset"],
io_manager_key="polars_parquet_io_manager",
metadata={
"partition_by": ["year", "month", "day"]
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_polars.PolarsDeltaIOManager'>dagster_polars.PolarsDeltaIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_polars/io_managers/delta.py#L68' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_polars.PolarsDeltaIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Implements writing and reading DeltaLake tables.
Features:
- All features provided by [`BasePolarsUPathIOManager`](#dagster_polars.BasePolarsUPathIOManager).
- All read/write options can be set via corresponding metadata or config parameters (metadata takes precedence).
- Supports native DeltaLake partitioning by storing different asset partitions in the same DeltaLake table. To enable this behavior, set the <cite>partition_by</cite> metadata value or config parameter <strong>and</strong> use a non-dict type annotation when loading the asset. The <cite>partition_by</cite> value will be used in <cite>delta_write_options</cite> of <cite>pl.DataFrame.write_delta</cite> and <cite>pyarrow_options</cite> of <cite>pl.scan_delta</cite>). When using a one-dimensional <cite>PartitionsDefinition</cite>, it should be a single string like <cite>“column”</cite>. When using a <cite>MultiPartitionsDefinition</cite>, it should be a dict with dimension to column names mapping, like <cite>\{“dimension”: “column”}</cite>.
Install <cite>dagster-polars[delta]</cite> to use this IOManager.
Examples:
```python
from dagster import asset
from dagster_polars import PolarsDeltaIOManager
import polars as pl
@asset(
io_manager_key="polars_delta_io_manager",
key_prefix=["my_dataset"]
)
def my_asset() -> pl.DataFrame: # data will be stored at <base_dir>/my_dataset/my_asset.delta
...
defs = Definitions(
assets=[my_table],
resources={
"polars_delta_io_manager": PolarsDeltaIOManager(base_dir="s3://my-bucket/my-dir")
}
)
```
Appending to a DeltaLake table and merging schema:
```python
@asset(
io_manager_key="polars_delta_io_manager",
metadata={
"mode": "append",
"delta_write_options": {"schema_mode":"merge"},
},
)
def my_table() -> pl.DataFrame:
...
```
Overwriting the schema if it has changed:
```python
@asset(
io_manager_key="polars_delta_io_manager",
metadata={
"mode": "overwrite",
"delta_write_options": {
"schema_mode": "overwrite"
},
)
def my_table() -> pl.DataFrame:
...
```
Using native DeltaLake partitioning by storing different asset partitions in the same DeltaLake table:
```python
from dagster import AssetExecutionContext, DailyPartitionedDefinition
from dagster_polars import LazyFramePartitions
@asset(
io_manager_key="polars_delta_io_manager",
metadata={
"partition_by": "partition_col"
},
partitions_def=StaticPartitionsDefinition(["a, "b", "c"])
)
def upstream(context: AssetExecutionContext) -> pl.DataFrame:
df = ...
# column with the partition_key must match `partition_by` metadata value
return df.with_columns(pl.lit(context.partition_key).alias("partition_col"))
@asset
def downstream(upstream: pl.LazyFrame) -> pl.DataFrame:
...
```
When using <cite>MuiltiPartitionsDefinition</cite>, <cite>partition_by</cite> metadata value should be a dictionary mapping dimensions to column names.
```python
from dagster import AssetExecutionContext, DailyPartitionedDefinition, MultiPartitionsDefinition, StaticPartitionsDefinition
from dagster_polars import LazyFramePartitions
@asset(
io_manager_key="polars_delta_io_manager",
metadata={
"partition_by": {"time": "date", "clients": "client"} # dimension -> column mapping
},
partitions_def=MultiPartitionsDefinition(
{
"date": DailyPartitionedDefinition(...),
"clients": StaticPartitionsDefinition(...)
}
)
)
def upstream(context: AssetExecutionContext) -> pl.DataFrame:
df = ...
partition_keys_by_dimension = context.partition_key.keys_by_dimension
return df.with_columns(
pl.lit(partition_keys_by_dimension["time"]).alias("date"), # time dimension matches date column
pl.lit(partition_keys_by_dimension["clients"]).alias("client") # clients dimension matches client column
)
@asset
def downstream(upstream: pl.LazyFrame) -> pl.DataFrame:
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_polars.PolarsBigQueryIOManager'>dagster_polars.PolarsBigQueryIOManager IOManagerDefinition <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_polars/io_managers/bigquery.py#L123' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_polars.PolarsBigQueryIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Implements reading and writing Polars DataFrames from/to [BigQuery](https://cloud.google.com/bigquery)).
Features:
- All `DBIOManager` features
- Supports writing partitioned tables (<cite>“partition_expr”</cite> input metadata key must be specified).
Returns: IOManagerDefinition
Examples:
```python
from dagster import Definitions, EnvVar
from dagster_polars import PolarsBigQueryIOManager
@asset(
key_prefix=["my_dataset"] # will be used as the dataset in BigQuery
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": PolarsBigQueryIOManager(project=EnvVar("GCP_PROJECT"))
}
)
```
You can tell Dagster in which dataset to create tables by setting the “dataset” configuration value.
If you do not provide a dataset as configuration to the I/O manager, Dagster will determine a dataset based
on the assets and ops using the I/O Manager. For assets, the dataset will be determined from the asset key,
as shown in the above example. The final prefix before the asset name will be used as the dataset. For example,
if the asset “my_table” had the key prefix [“gcp”, “bigquery”, “my_dataset”], the dataset “my_dataset” will be
used. For ops, the dataset can be specified by including a “schema” entry in output metadata. If “schema” is
not provided via config or on the asset/op, “public” will be used for the dataset.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_dataset"})}
)
def make_my_table() -> pl.DataFrame:
# the returned value will be stored at my_dataset.my_table
...
```
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
If you cannot upload a file to your Dagster deployment, or otherwise cannot
[authenticate with GCP](https://cloud.google.com/docs/authentication/provide-credentials-adc)
via a standard method, you can provide a service account key as the “gcp_credentials” configuration.
Dagster will store this key in a temporary file and set GOOGLE_APPLICATION_CREDENTIALS to point to the file.
After the run completes, the file will be deleted, and GOOGLE_APPLICATION_CREDENTIALS will be
unset. The key must be base64 encoded to avoid issues with newlines in the keys. You can retrieve
the base64 encoded key with this shell command: cat $GOOGLE_APPLICATION_CREDENTIALS | base64
The “write_disposition” metadata key can be used to set the <cite>write_disposition</cite> parameter
of <cite>bigquery.JobConfig</cite>. For example, set it to <cite>“WRITE_APPEND”</cite> to append to an existing table intead of
overwriting it.
Install <cite>dagster-polars[gcp]</cite> to use this IOManager.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_polars.patito.patito_model_to_dagster_type'>dagster_polars.patito.patito_model_to_dagster_type <a href='https://github.com/dagster-io/dagster/blob/master/docs/.tox/sphinx-mdx-vercel/lib/python3.11/site-packages/dagster_polars/patito.py#L53' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_polars.patito.patito_model_to_dagster_type" class="hash-link"></a></Link></dt>
<dd>
Convert patito model to dagster type checking.
Compatible with any IOManager. Logs Dagster metadata associated with
the Patito model, such as <cite>dagster/column_schema</cite>.
Parameters:
- <strong>model</strong> (<em>type</em><em>[</em><em>pt.Model</em><em>]</em>) – the Patito model.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Dagster Type name. Defaults to the model class name.
- <strong>description</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Dagster Type description. By default it references the model class name.
Returns: Dagster type with patito validation function.Return type: [DagsterType](../../../api/dagster/types.mdx#dagster.DagsterType)
Examples:
```python
import dagster as dg
import patito as pt
class MyTable(pt.Model):
col_1: str | None
col_2: int = pt.Field(unique=True)
@asset(
dagster_type=patito_model_to_dagster_type(MyTable),
io_manager_key="my_io_manager",
)
def my_asset() -> pl.DataFrame:
return pl.DataFrame({
"col_1": ['a'],
"col_2": [2],
})
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-postgres library'
sidebar_position: 1000
title_meta: 'dagster-postgres library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-postgres library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-postgres-library">
# dagster-postgres library
<dl>
<dt><Link class="anchor" id='dagster_postgres.PostgresEventLogStorage'>dagster_postgres.PostgresEventLogStorage `=` \<class 'dagster_postgres.event_log.event_log.PostgresEventLogStorage'> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-postgres/dagster_postgres/event_log/event_log.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_postgres.PostgresEventLogStorage" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Postgres-backed event log storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
To use Postgres for all of the components of your instance storage, you can add the following
block to your `dagster.yaml`:
dagster.yaml
```YAML
storage:
postgres:
postgres_db:
username: my_username
password: my_password
hostname: my_hostname
db_name: my_database
port: 5432
```
If you are configuring the different storage components separately and are specifically
configuring your event log storage to use Postgres, you can add a block such as the following
to your `dagster.yaml`:
dagster.yaml
```YAML
event_log_storage:
module: dagster_postgres.event_log
class: PostgresEventLogStorage
config:
postgres_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { db_name }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_postgres.PostgresRunStorage'>dagster_postgres.PostgresRunStorage `=` \<class 'dagster_postgres.run_storage.run_storage.PostgresRunStorage'> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/run_storage.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_postgres.PostgresRunStorage" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Postgres-backed run storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
To use Postgres for all of the components of your instance storage, you can add the following
block to your `dagster.yaml`:
dagster.yaml
```YAML
storage:
postgres:
postgres_db:
username: my_username
password: my_password
hostname: my_hostname
db_name: my_database
port: 5432
```
If you are configuring the different storage components separately and are specifically
configuring your run storage to use Postgres, you can add a block such as the following
to your `dagster.yaml`:
dagster.yaml
```YAML
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { db_name }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_postgres.PostgresScheduleStorage'>dagster_postgres.PostgresScheduleStorage `=` \<class 'dagster_postgres.schedule_storage.schedule_storage.PostgresScheduleStorage'> <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-postgres/dagster_postgres/schedule_storage/schedule_storage.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_postgres.PostgresScheduleStorage" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Postgres-backed run storage.
Users should not directly instantiate this class; it is instantiated by internal machinery when
`dagster-webserver` and `dagster-graphql` load, based on the values in the `dagster.yaml` file in
`$DAGSTER_HOME`. Configuration of this class should be done by setting values in that file.
To use Postgres for all of the components of your instance storage, you can add the following
block to your `dagster.yaml`:
dagster.yaml
```YAML
storage:
postgres:
postgres_db:
username: my_username
password: my_password
hostname: my_hostname
db_name: my_database
port: 5432
```
If you are configuring the different storage components separately and are specifically
configuring your schedule storage to use Postgres, you can add a block such as the following
to your `dagster.yaml`:
dagster.yaml
```YAML
schedule_storage:
module: dagster_postgres.schedule_storage
class: PostgresScheduleStorage
config:
postgres_db:
username: { username }
password: { password }
hostname: { hostname }
db_name: { db_name }
port: { port }
```
Note that the fields in this config are [`StringSource`](../../../api/dagster/config.mdx#dagster.StringSource) and
[`IntSource`](../../../api/dagster/config.mdx#dagster.IntSource) and can be configured from environment variables.
</dd>
</dl>
</div>
---
---
title: 'dagster-powerbi library'
sidebar_position: 1000
title_meta: 'dagster-powerbi library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-powerbi library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-powerbi-library">
# dagster-powerbi library
Dagster allows you to represent your Power BI Workspaces as assets, alongside other your other
technologies like dbt and Sling. This allows you to see how your Power BI assets are connected to
your other data assets, and how changes to other data assets might impact your Power BI Workspaces.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspaceComponent'>`class` dagster_powerbi.PowerBIWorkspaceComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/components/power_bi_workspace/component.py#L171' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspaceComponent" class="hash-link"></a></Link></dt>
<dd>
Pulls in the contents of a PowerBI workspace into Dagster assets.
Example:
```yaml
# defs.yaml
type: dagster_powerbi.PowerBIWorkspaceComponent
attributes:
workspace:
credentials:
client_id: "{{ env.POWERBI_CLIENT_ID }}"
client_secret: "{{ env.POWERBI_CLIENT_SECRET }}"
tenant_id: "{{ env.POWERBI_TENANT_ID }}"
workspace_id: your-workspace-id
enable_semantic_model_refresh: true
```
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspaceComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/components/power_bi_workspace/component.py#L222' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspaceComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Power BI content item.
This method can be overridden in a subclass to customize how Power BI content
(reports, dashboards, semantic models, datasets) are converted to Dagster asset specs.
By default, it delegates to the configured DagsterPowerBITranslator.
Parameters: <strong>data</strong> – The PowerBITranslatorData containing information about the Power BI content
item and workspaceReturns: An AssetSpec that represents the Power BI content as a Dagster asset
Example:
Override this method to add custom metadata based on content properties:
```python
from dagster_powerbi import PowerBIWorkspaceComponent
from dagster import AssetSpec
class CustomPowerBIWorkspaceComponent(PowerBIWorkspaceComponent):
def get_asset_spec(self, data):
base_spec = super().get_asset_spec(data)
return base_spec.replace_attributes(
metadata={
**base_spec.metadata,
"workspace_name": data.workspace_data.properties.get("name"),
"content_type": data.content_type
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Power BI component, see the [Power BI component integration guide](https://docs.dagster.io/integrations/libraries/powerbi).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Power BI component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_powerbi.PowerBIWorkspaceComponent
attributes:
workspace:
workspace_id: "{{ env.POWERBI_WORKSPACE_ID }}"
credentials:
client_id: "{{ env.POWERBI_CLIENT_ID }}"
client_secret: "{{ env.POWERBI_CLIENT_SECRET }}"
tenant_id: "{{ env.POWERBI_TENANT_ID }}"
# Alternatively, you can use an API access token
# credentials:
# token: "{{ env.POWERBI_API_TOKEN }}"
```
</div></div>
<div class="section" id="assets-power-bi-api">
## Assets (Power BI API)
Here, we provide interfaces to manage Power BI Workspaces using the Power BI API.
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIServicePrincipal'>`class` dagster_powerbi.PowerBIServicePrincipal <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L63' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIServicePrincipal" class="hash-link"></a></Link></dt>
<dd>
Authenticates with PowerBI using a service principal.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIToken'>`class` dagster_powerbi.PowerBIToken <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIToken" class="hash-link"></a></Link></dt>
<dd>
Authenticates with PowerBI directly using an API access token.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspace'>`class` dagster_powerbi.PowerBIWorkspace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L101' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspace" class="hash-link"></a></Link></dt>
<dd>
Represents a workspace in PowerBI and provides utilities
to interact with the PowerBI API.
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspace.poll_refresh'>poll_refresh <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L184' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspace.poll_refresh" class="hash-link"></a></Link></dt>
<dd>
Polls the refresh status of a PowerBI dataset until it completes or fails.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspace.trigger_and_poll_refresh'>trigger_and_poll_refresh <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L166' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspace.trigger_and_poll_refresh" class="hash-link"></a></Link></dt>
<dd>
Triggers a refresh of a PowerBI dataset and polls until it completes or fails.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.PowerBIWorkspace.trigger_refresh'>trigger_refresh <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L172' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.PowerBIWorkspace.trigger_refresh" class="hash-link"></a></Link></dt>
<dd>
Triggers a refresh of a PowerBI dataset.
</dd>
</dl>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.DagsterPowerBITranslator'>`class` dagster_powerbi.DagsterPowerBITranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/translator.py#L171' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.DagsterPowerBITranslator" class="hash-link"></a></Link></dt>
<dd>
Translator class which converts raw response data from the PowerBI API into AssetSpecs.
Subclass this class to implement custom logic for each type of PowerBI content.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.load_powerbi_asset_specs'>dagster_powerbi.load_powerbi_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/resource.py#L351' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.load_powerbi_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns a list of AssetSpecs representing the Power BI content in the workspace.
Parameters:
- <strong>workspace</strong> ([*PowerBIWorkspace*](#dagster_powerbi.PowerBIWorkspace)) – The Power BI workspace to load assets from.
- <strong>dagster_powerbi_translator</strong> (<em>Optional</em><em>[</em>[*DagsterPowerBITranslator*](#dagster_powerbi.DagsterPowerBITranslator)<em>]</em>) – The translator to use to convert Power BI content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterPowerBITranslator`](#dagster_powerbi.DagsterPowerBITranslator).
- <strong>use_workspace_scan</strong> (<em>bool</em>) – Whether to scan the entire workspace using admin APIs at once to get all content. Defaults to True.
Returns: The set of assets representing the Power BI content in the workspace.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_powerbi.build_semantic_model_refresh_asset_definition'>dagster_powerbi.build_semantic_model_refresh_asset_definition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-powerbi/dagster_powerbi/assets.py#L18' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_powerbi.build_semantic_model_refresh_asset_definition" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Builds an asset definition for refreshing a PowerBI semantic model.
</dd>
</dl>
</div></div>
---
---
title: 'dagster-prometheus library'
sidebar_position: 1000
title_meta: 'dagster-prometheus library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-prometheus library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-prometheus-library">
# dagster-prometheus library
<dl>
<dt><Link class="anchor" id='dagster_prometheus.PrometheusResource'>dagster_prometheus.PrometheusResource ResourceDefinition<a href="#dagster_prometheus.PrometheusResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
This resource is used to send metrics to a Prometheus Pushgateway.
<strong>Example:</strong>
```python
from dagster_prometheus import PrometheusResource
from dagster import Definitions, job, op
@op
def example_prometheus_op(prometheus: PrometheusResource):
prometheus.push_to_gateway(job="my_job")
@job
def my_job():
example_prometheus_op()
Definitions(
jobs=[my_job],
resources={"prometheus": PrometheusResource(gateway="http://pushgateway.local")},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_prometheus.resources.PrometheusClient'>`class` dagster_prometheus.resources.PrometheusClient<a href="#dagster_prometheus.resources.PrometheusClient" class="hash-link"></a></Link></dt>
<dd>
Integrates with Prometheus via the prometheus_client library.
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_prometheus.prometheus_resource'>dagster_prometheus.prometheus_resource ResourceDefinition<a href="#dagster_prometheus.prometheus_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
</dd>
</dl>
</div></div>
---
---
title: 'dagster-pyspark library'
sidebar_position: 1000
title_meta: 'dagster-pyspark library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-pyspark library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-pyspark-library">
# dagster-pyspark library
<dl>
<dt><Link class="anchor" id='dagster_pyspark.PySparkResource'>dagster_pyspark.PySparkResource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pyspark/dagster_pyspark/resources.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pyspark.PySparkResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource provides access to a PySpark Session for executing PySpark code within Dagster.
Example:
```python
@op
def my_op(pyspark: PySparkResource)
spark_session = pyspark.spark_session
dataframe = spark_session.read.json("examples/src/main/resources/people.json")
@job(
resource_defs={
"pyspark": PySparkResource(
spark_config={
"spark.executor.memory": "2g"
}
)
}
)
def my_spark_job():
my_op()
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_pyspark.pyspark_resource'>dagster_pyspark.pyspark_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-pyspark/dagster_pyspark/resources.py#L67' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_pyspark.pyspark_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource provides access to a PySpark SparkSession for executing PySpark code within Dagster.
Example:
```python
@op(required_resource_keys={"pyspark"})
def my_op(context):
spark_session = context.resources.pyspark.spark_session
dataframe = spark_session.read.json("examples/src/main/resources/people.json")
my_pyspark_resource = pyspark_resource.configured(
{"spark_conf": {"spark.executor.memory": "2g"}}
)
@job(resource_defs={"pyspark": my_pyspark_resource})
def my_spark_job():
my_op()
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-sigma library'
sidebar_position: 1000
title_meta: 'dagster-sigma library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-sigma library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-sigma-library">
# dagster-sigma library
Dagster allows you to represent the workbooks and datasets in your Sigma project as assets alongside other
technologies including dbt and Sling. This allows you to visualize relationships between your Sigma assets
and their dependencies.
Related documentation pages: [Using Dagster with Sigma](https://docs.dagster.io/integrations/libraries/sigma).
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaComponent'>`class` dagster_sigma.SigmaComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/components/sigma_component.py#L176' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaComponent" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Pulls in the contents of a Sigma organization into Dagster assets.
Example:
```yaml
# defs.yaml
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: https://aws-api.sigmacomputing.com
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
sigma_filter:
workbook_folders:
- ["My Documents", "Analytics"]
include_unused_datasets: false
```
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/components/sigma_component.py#L249' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Sigma content item.
This method can be overridden in a subclass to customize how Sigma content
(workbooks, datasets) are converted to Dagster asset specs. By default, it delegates
to the configured DagsterSigmaTranslator.
Parameters: <strong>data</strong> – The SigmaTranslatorData containing information about the Sigma content item
and organizationReturns: An AssetSpec that represents the Sigma content as a Dagster asset
Example:
Override this method to add custom tags based on content properties:
```python
from dagster_sigma import SigmaComponent
from dagster import AssetSpec
class CustomSigmaComponent(SigmaComponent):
def get_asset_spec(self, data):
base_spec = super().get_asset_spec(data)
return base_spec.replace_attributes(
tags={
**base_spec.tags,
"sigma_type": data.properties.get("type"),
"owner": data.properties.get("ownerId")
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Sigma component, see the [Sigma component integration guide](https://docs.dagster.io/integrations/libraries/sigma).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Sigma component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: "{{ env.SIGMA_BASE_URL }}"
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
```
</div></div>
<div class="section" id="sigma-api">
## Sigma API
Here, we provide interfaces to manage Sigma projects using the Sigma API.
<div class="section" id="assets-sigma-api">
### Assets (Sigma API)
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaOrganization'>`class` dagster_sigma.SigmaOrganization <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/resource.py#L129' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaOrganization" class="hash-link"></a></Link></dt>
<dd>
Represents a workspace in Sigma and provides utilities
to interact with the Sigma API.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaBaseUrl'>`class` dagster_sigma.SigmaBaseUrl <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/resource.py#L115' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaBaseUrl" class="hash-link"></a></Link></dt>
<dd>
Enumeration of Sigma API base URLs for different cloud providers.
[https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.DagsterSigmaTranslator'>`class` dagster_sigma.DagsterSigmaTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/translator.py#L160' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.DagsterSigmaTranslator" class="hash-link"></a></Link></dt>
<dd>
Translator class which converts raw response data from the Sigma API into AssetSpecs.
Subclass this class to provide custom translation logic.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaDataset'>`class` dagster_sigma.SigmaDataset <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/translator.py#L64' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaDataset" class="hash-link"></a></Link></dt>
<dd>
Represents a Sigma dataset, a centralized data definition which can
contain aggregations or other manipulations.
[https://help.sigmacomputing.com/docs/datasets](https://help.sigmacomputing.com/docs/datasets)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaWorkbook'>`class` dagster_sigma.SigmaWorkbook <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/translator.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaWorkbook" class="hash-link"></a></Link></dt>
<dd>
Represents a Sigma workbook, a collection of visualizations and queries
for data exploration and analysis.
[https://help.sigmacomputing.com/docs/workbooks](https://help.sigmacomputing.com/docs/workbooks)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.SigmaFilter'>`class` dagster_sigma.SigmaFilter <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/resource.py#L79' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.SigmaFilter" class="hash-link"></a></Link></dt>
<dd>
Filters the set of Sigma objects to fetch.
Parameters:
- <strong>workbook_folders</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – A list of folder paths to fetch workbooks from. Each folder path is a list of folder names, starting from the root folder. All workbooks contained in the specified folders will be fetched. If not provided, all workbooks will be fetched.
- <strong>workbooks</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em><em>]</em>) – A list of fully qualified workbook paths to fetch. Each workbook path is a list of folder names, starting from the root folder, and ending with the workbook name. If not provided, all workbooks will be fetched.
- <strong>include_unused_datasets</strong> (<em>bool</em>) – Whether to include datasets that are not used in any workbooks. Defaults to True.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sigma.load_sigma_asset_specs'>dagster_sigma.load_sigma_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sigma/dagster_sigma/resource.py#L696' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sigma.load_sigma_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns a list of AssetSpecs representing the Sigma content in the organization.
Parameters:
- <strong>organization</strong> ([*SigmaOrganization*](#dagster_sigma.SigmaOrganization)) – The Sigma organization to fetch assets from.
- <strong>dagster_sigma_translator</strong> (<em>Optional</em><em>[</em>[*DagsterSigmaTranslator*](#dagster_sigma.DagsterSigmaTranslator)<em>]</em>) – The translator to use to convert Sigma content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterSigmaTranslator`](#dagster_sigma.DagsterSigmaTranslator).
- <strong>sigma_filter</strong> (<em>Optional</em><em>[</em>[*SigmaFilter*](#dagster_sigma.SigmaFilter)<em>]</em>) – Filters the set of Sigma objects to fetch.
- <strong>fetch_column_data</strong> (<em>bool</em>) – Whether to fetch column data for datasets, which can be slow.
- <strong>fetch_lineage_data</strong> (<em>bool</em>) – Whether to fetch any lineage data for workbooks and datasets.
- <strong>snapshot_path</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>str</em><em>, </em><em>Path</em><em>]</em><em>]</em>) – Path to a snapshot file to load Sigma data from, rather than fetching it from the Sigma API.
Returns: The set of assets representing the Sigma content in the organization.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-slack library'
sidebar_position: 1000
title_meta: 'dagster-slack library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-slack library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-slack-library">
# dagster-slack library
<div className='lineblock'> </div>
This library provides an integration with Slack, to support posting messages in your company’s Slack workspace.
<div className='lineblock'> </div>
Presently, it provides a thin wrapper on the Slack client API [chat.postMessage](https://api.slack.com/methods/chat.postMessage).
<div className='lineblock'> </div>
To use this integration, you’ll first need to create a Slack App for it.
1. <strong>Create App</strong>: Go to [https://api.slack.com/apps](https://api.slack.com/apps) and click “Create New App”:
2. <strong>Install App</strong>: After creating an app, on the left-hand side of the app configuration, click “Bot Users”, and then create a bot user. Then, click “Install App” on the left hand side, and finally “Install App to Workspace”.
3. <strong>Bot Token</strong>: Once finished, this will create a new bot token for your bot/workspace:
Copy this bot token and put it somewhere safe; see [Safely Storing Credentials](https://api.slack.com/docs/oauth-safety) for more on this topic.
<dl>
<dt><Link class="anchor" id='dagster_slack.SlackResource'>dagster_slack.SlackResource ResourceDefinition<a href="#dagster_slack.SlackResource" class="hash-link"></a></Link></dt>
<dd>
This resource is for connecting to Slack.
By configuring this Slack resource, you can post messages to Slack from any Dagster op, asset, schedule or sensor.
Examples:
```python
import os
from dagster import EnvVar, job, op
from dagster_slack import SlackResource
@op
def slack_op(slack: SlackResource):
slack.get_client().chat_postMessage(channel='#noise', text=':wave: hey there!')
@job
def slack_job():
slack_op()
Definitions(
jobs=[slack_job],
resources={
"slack": SlackResource(token=EnvVar("MY_SLACK_TOKEN")),
},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_slack.make_slack_on_run_failure_sensor'>dagster_slack.make_slack_on_run_failure_sensor<a href="#dagster_slack.make_slack_on_run_failure_sensor" class="hash-link"></a></Link></dt>
<dd>
Create a sensor on job failures that will message the given Slack channel.
Parameters:
- <strong>channel</strong> (<em>str</em>) – The channel to send the message to (e.g. “#my_channel”)
- <strong>slack_token</strong> (<em>str</em>) – The slack token. Tokens are typically either user tokens or bot tokens. More in the Slack API documentation here: [https://api.slack.com/docs/token-types](https://api.slack.com/docs/token-types)
- <strong>text_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*RunFailureSensorContext*](../../../api/dagster/schedules-sensors.mdx#dagster.RunFailureSensorContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the `RunFailureSensorContext` and outputs the message you want to send. Defaults to a text message that contains error message, job name, and run ID. The usage of the <cite>text_fn</cite> changes depending on whether you’re using <cite>blocks_fn</cite>. If you are using <cite>blocks_fn</cite>, this is used as a fallback string to display in notifications. If you aren’t, this is the main body text of the message. It can be formatted as plain text, or with markdown. See more details in [https://api.slack.com/methods/chat.postMessage#text_usage](https://api.slack.com/methods/chat.postMessage#text_usage)
- <strong>blocks_fn</strong> (<em>Callable</em><em>[</em><em>[</em>[*RunFailureSensorContext*](../../../api/dagster/schedules-sensors.mdx#dagster.RunFailureSensorContext)<em>]</em><em>, </em><em>List</em><em>[</em><em>Dict</em><em>]</em><em>]</em>) – Function which takes in the `RunFailureSensorContext` and outputs the message blocks you want to send. See information about Blocks in [https://api.slack.com/reference/block-kit/blocks](https://api.slack.com/reference/block-kit/blocks)
- <strong>name</strong> – (Optional[str]): The name of the sensor. Defaults to “slack_on_run_failure”.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your Dagit instance. Specify this to allow messages to include deeplinks to the failed job run.
- <strong>minimum_interval_seconds</strong> – (Optional[int]): The minimum number of seconds that will elapse between sensor evaluations.
- <strong>monitored_jobs</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](../../../api/dagster/jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](../../../api/dagster/graphs.mdx#dagster.GraphDefinition)<em>, </em>[*RepositorySelector*](../../../api/dagster/schedules-sensors.mdx#dagster.RepositorySelector)<em>, </em>[*JobSelector*](../../../api/dagster/schedules-sensors.mdx#dagster.JobSelector)<em>, </em><em>CodeLocationSensor</em><em>]</em><em>]</em><em>]</em>) – The jobs in the current repository that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the repository fails. To monitor jobs in external repositories, use RepositorySelector and JobSelector
- <strong>job_selection</strong> (<em>Optional</em><em>[</em><em>List</em><em>[</em><em>Union</em><em>[</em>[*JobDefinition*](../../../api/dagster/jobs.mdx#dagster.JobDefinition)<em>, </em>[*GraphDefinition*](../../../api/dagster/graphs.mdx#dagster.GraphDefinition)<em>, </em>[*RepositorySelector*](../../../api/dagster/schedules-sensors.mdx#dagster.RepositorySelector)<em>, </em>[*JobSelector*](../../../api/dagster/schedules-sensors.mdx#dagster.JobSelector)<em>, </em><em>CodeLocationSensor</em><em>]</em><em>]</em><em>]</em>) – <span className="flag flag-warning">deprecated</span> (deprecated in favor of monitored_jobs) The jobs in the current repository that will be monitored by this failure sensor. Defaults to None, which means the alert will be sent when any job in the repository fails.
- <strong>monitor_all_code_locations</strong> (<em>bool</em>) – If set to True, the sensor will monitor all runs in the Dagster deployment. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
- <strong>default_status</strong> (<em>DefaultSensorStatus</em>) – Whether the sensor starts as running or not. The default status can be overridden from Dagit or via the GraphQL API.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the failed job run.
- <strong>monitor_all_repositories</strong> (<em>bool</em>) – <span className="flag flag-warning">deprecated</span> If set to True, the sensor will monitor all runs in the Dagster instance. If set to True, an error will be raised if you also specify monitored_jobs or job_selection. Defaults to False.
Examples:
```python
slack_on_run_failure = make_slack_on_run_failure_sensor(
"#my_channel",
os.getenv("MY_SLACK_TOKEN")
)
@repository
def my_repo():
return [my_job + slack_on_run_failure]
```
```python
def my_message_fn(context: RunFailureSensorContext) -> str:
return (
f"Job {context.dagster_run.job_name} failed!"
f"Error: {context.failure_event.message}"
)
slack_on_run_failure = make_slack_on_run_failure_sensor(
channel="#my_channel",
slack_token=os.getenv("MY_SLACK_TOKEN"),
text_fn=my_message_fn,
webserver_base_url="http://mycoolsite.com",
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_slack.slack_on_failure'>dagster_slack.slack_on_failure HookDefinition<a href="#dagster_slack.slack_on_failure" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step failure events that will message the given Slack channel.
Parameters:
- <strong>channel</strong> (<em>str</em>) – The channel to send the message to (e.g. “#my_channel”)
- <strong>message_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*HookContext*](../../../api/dagster/hooks.mdx#dagster.HookContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the HookContext outputs the message you want to send.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
Examples:
```python
@slack_on_failure("#foo", webserver_base_url="http://localhost:3000")
@job(...)
def my_job():
pass
```
```python
def my_message_fn(context: HookContext) -> str:
return f"Op {context.op} failed!"
@op
def an_op(context):
pass
@job(...)
def my_job():
an_op.with_hooks(hook_defs={slack_on_failure("#foo", my_message_fn)})
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_slack.slack_on_success'>dagster_slack.slack_on_success HookDefinition<a href="#dagster_slack.slack_on_success" class="hash-link"></a></Link></dt>
<dd>
Create a hook on step success events that will message the given Slack channel.
Parameters:
- <strong>channel</strong> (<em>str</em>) – The channel to send the message to (e.g. “#my_channel”)
- <strong>message_fn</strong> (<em>Optional</em><em>(</em><em>Callable</em><em>[</em><em>[</em>[*HookContext*](../../../api/dagster/hooks.mdx#dagster.HookContext)<em>]</em><em>, </em><em>str</em><em>]</em><em>)</em>) – Function which takes in the HookContext outputs the message you want to send.
- <strong>dagit_base_url</strong> – <span className="flag flag-warning">deprecated</span> (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
- <strong>webserver_base_url</strong> – (Optional[str]): The base url of your webserver instance. Specify this to allow messages to include deeplinks to the specific run that triggered the hook.
Examples:
```python
@slack_on_success("#foo", webserver_base_url="http://localhost:3000")
@job(...)
def my_job():
pass
```
```python
def my_message_fn(context: HookContext) -> str:
return f"Op {context.op} worked!"
@op
def an_op(context):
pass
@job(...)
def my_job():
an_op.with_hooks(hook_defs={slack_on_success("#foo", my_message_fn)})
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_slack.slack_resource'>dagster_slack.slack_resource ResourceDefinition<a href="#dagster_slack.slack_resource" class="hash-link"></a></Link></dt>
<dd>
This resource is for connecting to Slack.
The resource object is a <cite>slack_sdk.WebClient</cite>.
By configuring this Slack resource, you can post messages to Slack from any Dagster op, asset, schedule or sensor.
Examples:
```python
import os
from dagster import job, op
from dagster_slack import slack_resource
@op(required_resource_keys={'slack'})
def slack_op(context):
context.resources.slack.chat_postMessage(channel='#noise', text=':wave: hey there!')
@job(resource_defs={'slack': slack_resource})
def slack_job():
slack_op()
slack_job.execute_in_process(
run_config={'resources': {'slack': {'config': {'token': os.getenv('SLACK_TOKEN')}}}}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-sling library'
sidebar_position: 1000
title_meta: 'dagster-sling library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-sling library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-sling-library">
# dagster-sling library
This library provides a Dagster integration with [Sling](https://slingdata.io).
For more information on getting started, see the [Dagster & Sling](https://docs.dagster.io/integrations/libraries/sling) documentation.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_sling.SlingReplicationCollectionComponent'>`class` dagster_sling.SlingReplicationCollectionComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/components/sling_replication_collection/component.py#L117' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.SlingReplicationCollectionComponent" class="hash-link"></a></Link></dt>
<dd>
Expose one or more Sling replications to Dagster as assets.
To get started, run:
`dg scaffold defs dagster_sling.SlingReplicationCollectionComponent \{defs_path}`
This will create a defs.yaml as well as a `replication.yaml`, which is a Sling-specific configuration
file. See Sling’s [documentation](https://docs.slingdata.io/concepts/replication#overview) on `replication.yaml`.
<dl>
<dt><Link class="anchor" id='dagster_sling.SlingReplicationCollectionComponent.execute'>execute <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/components/sling_replication_collection/component.py#L204' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.SlingReplicationCollectionComponent.execute" class="hash-link"></a></Link></dt>
<dd>
Executes a Sling replication for the selected streams.
This method can be overridden in a subclass to customize the replication execution
behavior, such as adding custom logging, modifying metadata collection, or handling
results differently.
Parameters:
- <strong>context</strong> – The asset execution context provided by Dagster
- <strong>sling</strong> – The SlingResource used to execute the replication
- <strong>replication_spec_model</strong> – The model containing replication configuration and metadata options
Yields: AssetMaterialization or MaterializeResult events from the Sling replication
Example:
Override this method to add custom logging during replication:
```python
from dagster_sling import SlingReplicationCollectionComponent
from dagster import AssetExecutionContext
class CustomSlingComponent(SlingReplicationCollectionComponent):
def execute(self, context, sling, replication_spec_model):
context.log.info("Starting Sling replication")
yield from super().execute(context, sling, replication_spec_model)
context.log.info("Sling replication completed")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.SlingReplicationCollectionComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/components/sling_replication_collection/component.py#L146' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.SlingReplicationCollectionComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Sling stream definition.
This method can be overridden in a subclass to customize how Sling stream definitions
are converted to Dagster asset specs. By default, it delegates to the configured
DagsterSlingTranslator.
Parameters: <strong>stream_definition</strong> – A dictionary representing a single stream from the Sling
replication config, containing source and target informationReturns: An AssetSpec that represents the Sling stream as a Dagster asset
Example:
Override this method to add custom metadata based on stream properties:
```python
from dagster_sling import SlingReplicationCollectionComponent
from dagster import AssetSpec
class CustomSlingComponent(SlingReplicationCollectionComponent):
def get_asset_spec(self, stream_definition):
base_spec = super().get_asset_spec(stream_definition)
return base_spec.replace_attributes(
metadata={
**base_spec.metadata,
"source": stream_definition.get("source"),
"target": stream_definition.get("target")
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Sling component, see the [Sling component integration guide](https://docs.dagster.io/integrations/libraries/sling).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Sling component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
replications:
- path: replication.yaml
```
The following `replication.yaml` file will also be created:
```yaml
source: {}
streams: {}
target: {}
```
</div></div>
<div class="section" id="assets">
## Assets
<dl>
<dt><Link class="anchor" id='dagster_sling.sling_assets'>@dagster_sling.sling_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/asset_decorator.py#L55' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.sling_assets" class="hash-link"></a></Link></dt>
<dd>
Create a definition for how to materialize a set of Sling replication streams as Dagster assets, as
described by a Sling replication config. This will create on Asset for every Sling target stream.
A Sling Replication config is a configuration that maps sources to destinations. For the full
spec and descriptions, see [Sling’s Documentation](https://docs.slingdata.io/sling-cli/run/configuration).
Parameters:
- <strong>replication_config</strong> (<em>Union</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>, </em><em>str</em><em>, </em><em>Path</em><em>]</em>) – A path to a Sling replication config, or a dictionary of a replication config.
- <strong>dagster_sling_translator</strong> – (DagsterSlingTranslator): Allows customization of how to map a Sling stream to a Dagster AssetKey.
- <strong>(</strong><strong>Optional</strong><strong>[</strong><strong>str</strong><strong>]</strong> (<em>name</em>) – The name of the op.
- <strong>partitions_def</strong> (<em>Optional</em><em>[</em>[*PartitionsDefinition*](../../../api/dagster/partitions.mdx#dagster.PartitionsDefinition)<em>]</em>) – The partitions definition for this asset.
- <strong>backfill_policy</strong> (<em>Optional</em><em>[</em>[*BackfillPolicy*](../../../api/dagster/partitions.mdx#dagster.BackfillPolicy)<em>]</em>) – The backfill policy for this asset.
- <strong>op_tags</strong> (<em>Optional</em><em>[</em><em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em><em>]</em>) – The tags for the underlying op.
- <strong>pool</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – A string that identifies the concurrency pool that governs the sling assets’ execution.
Examples:
Running a sync by providing a path to a Sling Replication config:
```python
from dagster_sling import sling_assets, SlingResource, SlingConnectionResource
sling_resource = SlingResource(
connections=[
SlingConnectionResource(
name="MY_POSTGRES", type="postgres", connection_string=EnvVar("POSTGRES_URL")
),
SlingConnectionResource(
name="MY_DUCKDB",
type="duckdb",
connection_string="duckdb:///var/tmp/duckdb.db",
),
]
)
config_path = "/path/to/replication.yaml"
@sling_assets(replication_config=config_path)
def my_assets(context, sling: SlingResource):
yield from sling.replicate(context=context)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator'>`class` dagster_sling.DagsterSlingTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator" class="hash-link"></a></Link></dt>
<dd>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_asset_key'>get_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L94' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_asset_key" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).key` instead..
:::
A function that takes a stream definition from a Sling replication config and returns a
Dagster AssetKey.
The stream definition is a dictionary key/value pair where the key is the stream name and
the value is a dictionary representing the Sling Replication Stream Config.
For example:
```python
stream_definition = {"public.users":
{'sql': 'select all_user_id, name from public."all_Users"',
'object': 'public.all_users'}
}
```
By default, this returns the class’s target_prefix parameter concatenated with the stream name.
A stream named “public.accounts” will create an AssetKey named “target_public_accounts”.
Override this function to customize how to map a Sling stream to a Dagster AssetKey.
Alternatively, you can provide metadata in your Sling replication config to specify the
Dagster AssetKey for a stream as follows:
```yaml
public.users:
meta:
dagster:
asset_key: "mydb_users"
```
Parameters: <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definitionReturns: The Dagster AssetKey for the replication stream.Return type: [AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)
Examples:
Using a custom mapping for streams:
```python
class CustomSlingTranslator(DagsterSlingTranslator):
def get_asset_spec(self, stream_definition: Mapping[str, Any]) -> AssetKey:
default_spec = super().get_asset_spec(stream_definition)
map = {"stream1": "asset1", "stream2": "asset2"}
return default_spec.replace_attributes(key=AssetKey(map[stream_definition["name"]]))
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
A function that takes a stream definition from a Sling replication config and returns a
Dagster AssetSpec.
The stream definition is a dictionary key/value pair where the key is the stream name and
the value is a dictionary representing the Sling Replication Stream Config.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_auto_materialize_policy'>get_auto_materialize_policy <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L465' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_auto_materialize_policy" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).auto_materialize_policy` instead..
:::
Defines the auto-materialize policy for a given stream definition.
This method checks the provided stream definition for a specific configuration
indicating an auto-materialize policy. If the configuration is found, it returns
an eager auto-materialize policy. Otherwise, it returns None.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: An eager auto-materialize policy if the configuration
is found, otherwise None.Return type: Optional[AutoMaterializePolicy]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_deps_asset_key'>get_deps_asset_key <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L212' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_deps_asset_key" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Iterate over `DagsterSlingTranslator.get_asset_spec(...).deps` to access `AssetDep.asset_key` instead..
:::
A function that takes a stream definition from a Sling replication config and returns a
Dagster AssetKey for each dependency of the replication stream.
By default, this returns the stream name. For example, a stream named “public.accounts”
will create an AssetKey named “target_public_accounts” and a dependency named “public_accounts”.
Override this function to customize how to map a Sling stream to a Dagster dependency.
Alternatively, you can provide metadata in your Sling replication config to specify the
Dagster AssetKey for a stream as follows:
```yaml
public.users:
meta:
dagster:
deps: "sourcedb_users"
```
Parameters: <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definitionReturns: A list of Dagster AssetKey for each dependency of the replication stream.Return type: Iterable[[AssetKey](../../../api/dagster/assets.mdx#dagster.AssetKey)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_description'>get_description <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L288' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_description" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).description` instead..
:::
Retrieves the description for a given stream definition.
This method checks the provided stream definition for a description. It first looks
for an “sql” key in the configuration and returns its value if found. If not, it looks
for a description in the metadata under the “dagster” key.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: The description of the stream if found, otherwise None.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_group_name'>get_group_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L429' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_group_name" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).group_name` instead..
:::
Retrieves the group name for a given stream definition.
This method checks the provided stream definition for a group name in the metadata
under the “dagster” key.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: The group name if found, otherwise None.Return type: Optional[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_kinds'>get_kinds <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L397' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_kinds" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).kinds` instead..
:::
Retrieves the kinds for a given stream definition.
This method returns “sling” by default. This method can be overridden to provide custom kinds.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: A set containing kinds for the stream’s assets.Return type: Set[str]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_metadata'>get_metadata <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L329' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_metadata" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).metadata` instead..
:::
Retrieves the metadata for a given stream definition.
This method extracts the configuration from the provided stream definition and returns
it as a JSON metadata value.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: A dictionary containing the stream configuration as JSON metadata.Return type: Mapping[str, Any]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.get_tags'>get_tags <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L363' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.get_tags" class="hash-link"></a></Link></dt>
<dd>
:::warning superseded
This API has been superseded.
Use `DagsterSlingTranslator.get_asset_spec(...).tags` instead..
:::
Retrieves the tags for a given stream definition.
This method returns an empty dictionary, indicating that no tags are associated with
the stream definition by default. This method can be overridden to provide custom tags.
Parameters:
- <strong>stream_definition</strong> (<em>Mapping</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>]</em>) – A dictionary representing the stream definition,
- <strong>details.</strong> (<em>which includes configuration</em>)
Returns: An empty dictionary.Return type: Mapping[str, Any]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.DagsterSlingTranslator.sanitize_stream_name'>sanitize_stream_name <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/dagster_sling_translator.py#L72' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.DagsterSlingTranslator.sanitize_stream_name" class="hash-link"></a></Link></dt>
<dd>
A function that takes a stream name from a Sling replication config and returns a
sanitized name for the stream.
By default, this removes any non-alphanumeric characters from the stream name and replaces
them with underscores, while removing any double quotes.
Parameters: <strong>stream_name</strong> (<em>str</em>) – The name of the stream.
Examples:
Using a custom stream name sanitizer:
```python
class CustomSlingTranslator(DagsterSlingTranslator):
def sanitize_stream_name(self, stream_name: str) -> str:
return stream_name.replace(".", "")
```
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_sling.SlingResource'>`class` dagster_sling.SlingResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/resources.py#L115' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.SlingResource" class="hash-link"></a></Link></dt>
<dd>
Resource for interacting with the Sling package. This resource can be used to run Sling replications.
Parameters: <strong>connections</strong> (<em>List</em><em>[</em>[*SlingConnectionResource*](#dagster_sling.SlingConnectionResource)<em>]</em>) – A list of connections to use for the replication.
Examples:
```python
from dagster_etl.sling import SlingResource, SlingConnectionResource
sling_resource = SlingResource(
connections=[
SlingConnectionResource(
name="MY_POSTGRES",
type="postgres",
connection_string=EnvVar("POSTGRES_CONNECTION_STRING"),
),
SlingConnectionResource(
name="MY_SNOWFLAKE",
type="snowflake",
host=EnvVar("SNOWFLAKE_HOST"),
user=EnvVar("SNOWFLAKE_USER"),
database=EnvVar("SNOWFLAKE_DATABASE"),
private_key=EnvVar("SNOWFLAKE_PRIVATE_KEY"),
role=EnvVar("SNOWFLAKE_ROLE"),
),
]
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_sling.SlingConnectionResource'>`class` dagster_sling.SlingConnectionResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-sling/dagster_sling/resources.py#L57' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_sling.SlingConnectionResource" class="hash-link"></a></Link></dt>
<dd>
A representation of a connection to a database or file to be used by Sling. This resource can be used as a source or a target for a Sling syncs.
Reference the Sling docs for more information on possible connection types and parameters: [https://docs.slingdata.io/connections](https://docs.slingdata.io/connections)
The name of the connection is passed to Sling and must match the name of the connection provided in the replication configuration: [https://docs.slingdata.io/sling-cli/run/configuration/replication](https://docs.slingdata.io/sling-cli/run/configuration/replication)
You may provide either a connection string or keyword arguments for the connection.
Examples:
Creating a Sling Connection for a file, such as CSV or JSON:
```python
source = SlingConnectionResource(name="MY_FILE", type="file")
```
Create a Sling Connection for a Postgres database, using a connection string:
```python
postgres_conn = SlingConnectionResource(name="MY_POSTGRES", type="postgres", connection_string=EnvVar("POSTGRES_CONNECTION_STRING"))
mysql_conn = SlingConnectionResource(name="MY_MYSQL", type="mysql", connection_string="mysql://user:password@host:port/schema")
```
Create a Sling Connection for a Postgres or Snowflake database, using keyword arguments:
</dd>
</dl>
</div></div>
---
---
title: 'dagster-snowflake-pandas library'
sidebar_position: 1000
title_meta: 'dagster-snowflake-pandas library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-snowflake-pandas library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-snowflake-pandas-library">
# dagster-snowflake-pandas library
This library provides an integration with the [Snowflake](https://www.snowflake.com) data
warehouse and Pandas data processing library.
To use this library, you should first ensure that you have an appropriate [Snowflake user](https://docs.snowflake.net/manuals/user-guide/admin-user-management.html) configured to access
your data warehouse.
Related Guides:
- [Using Dagster with Snowflake guides](https://docs.dagster.io/integrations/libraries/snowflake)
- [Snowflake I/O manager reference](https://docs.dagster.io/integrations/libraries/snowflake/reference)
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pandas.SnowflakePandasIOManager'>dagster_snowflake_pandas.SnowflakePandasIOManager IOManagerDefinition<a href="#dagster_snowflake_pandas.SnowflakePandasIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes Pandas DataFrames to Snowflake. When
using the SnowflakePandasIOManager, any inputs and outputs without type annotations will be loaded
as Pandas DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_pandas import SnowflakePandasIOManager
from dagster import asset, Definitions, EnvVar
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": SnowflakePandasIOManager(database="MY_DATABASE", account=EnvVar("SNOWFLAKE_ACCOUNT"))
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={
"io_manager": SnowflakePandasIOManager(database="my_database", schema="my_schema")
}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pandas.SnowflakePandasTypeHandler'>`class` dagster_snowflake_pandas.SnowflakePandasTypeHandler<a href="#dagster_snowflake_pandas.SnowflakePandasTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Plugin for the Snowflake I/O Manager that can store and load Pandas DataFrames as Snowflake tables.
Examples:
```python
from dagster_snowflake import SnowflakeIOManager
from dagster_snowflake_pandas import SnowflakePandasTypeHandler
from dagster_snowflake_pyspark import SnowflakePySparkTypeHandler
from dagster import Definitions, EnvVar
class MySnowflakeIOManager(SnowflakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [SnowflakePandasTypeHandler(), SnowflakePySparkTypeHandler()]
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": MySnowflakeIOManager(database="MY_DATABASE", account=EnvVar("SNOWFLAKE_ACCOUNT"), ...)
}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pandas.snowflake_pandas_io_manager'>dagster_snowflake_pandas.snowflake_pandas_io_manager IOManagerDefinition<a href="#dagster_snowflake_pandas.snowflake_pandas_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes Pandas DataFrames to Snowflake. When
using the snowflake_pandas_io_manager, any inputs and outputs without type annotations will be loaded
as Pandas DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_pandas import snowflake_pandas_io_manager
from dagster import asset, Definitions
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": snowflake_pandas_io_manager.configured({
"database": "my_database",
"account": {"env": "SNOWFLAKE_ACCOUNT"}
})
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table],
resources={"io_manager": snowflake_pandas_io_manager.configured(
{"database": "my_database", "schema": "my_schema"} # will be used as the schema
)}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-snowflake-polars library'
sidebar_position: 1000
title_meta: 'dagster-snowflake-polars library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-snowflake-polars library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-snowflake-polars-library">
# dagster-snowflake-polars library
This library provides an integration with Snowflake and Polars, allowing you to use Polars DataFrames with Snowflake storage.
<div class="section" id="i-o-manager">
## I/O manager
<dl>
<dt><Link class="anchor" id='dagster_snowflake_polars.snowflake_polars_io_manager'>dagster_snowflake_polars.snowflake_polars_io_manager<a href="#dagster_snowflake_polars.snowflake_polars_io_manager" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
An I/O manager definition that reads inputs from and writes Polars DataFrames to Snowflake. When
using the snowflake_polars_io_manager, any inputs and outputs without type annotations will be loaded
as Polars DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_polars import snowflake_polars_io_manager
from dagster import asset, Definitions
import polars as pl
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": snowflake_polars_io_manager.configured({
"database": "my_database",
"account": {"env": "SNOWFLAKE_ACCOUNT"}
})
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
defs = Definitions(
assets=[my_table],
resources={"io_manager": snowflake_polars_io_manager.configured(
{"database": "my_database", "schema": "my_schema"} # will be used as the schema
)}
)
```
On individual assets, you can also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pl.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pl.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pl.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame) -> pl.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake_polars.SnowflakePolarsIOManager'>`class` dagster_snowflake_polars.SnowflakePolarsIOManager<a href="#dagster_snowflake_polars.SnowflakePolarsIOManager" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
An I/O manager definition that reads inputs from and writes Polars DataFrames to Snowflake. When
using the SnowflakePolarsIOManager, any inputs and outputs without type annotations will be loaded
as Polars DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_polars import SnowflakePolarsIOManager
from dagster import asset, Definitions, EnvVar
import polars as pl
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": SnowflakePolarsIOManager(database="MY_DATABASE", account=EnvVar("SNOWFLAKE_ACCOUNT"))
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
defs = Definitions(
assets=[my_table],
resources={
"io_manager": SnowflakePolarsIOManager(database="my_database", schema="my_schema")
}
)
```
On individual assets, you can also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pl.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pl.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pl.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pl.DataFrame) -> pl.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div>
<div class="section" id="type-handler">
## Type handler
<dl>
<dt><Link class="anchor" id='dagster_snowflake_polars.SnowflakePolarsTypeHandler'>`class` dagster_snowflake_polars.SnowflakePolarsTypeHandler<a href="#dagster_snowflake_polars.SnowflakePolarsTypeHandler" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Plugin for the Snowflake I/O Manager that can store and load Polars DataFrames as Snowflake tables.
This handler uses Polars’ native write_database method with ADBC (Arrow Database Connectivity)
for efficient data transfer without converting to pandas.
Examples:
```python
from dagster_snowflake import SnowflakeIOManager
from dagster_snowflake_polars import SnowflakePolarsTypeHandler
from dagster import Definitions, EnvVar
class MySnowflakeIOManager(SnowflakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [SnowflakePolarsTypeHandler()]
@asset(
key_prefix=["my_schema"], # will be used as the schema in snowflake
)
def my_table() -> pl.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": MySnowflakeIOManager(database="MY_DATABASE", account=EnvVar("SNOWFLAKE_ACCOUNT"), ...)
}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-snowflake-pyspark library'
sidebar_position: 1000
title_meta: 'dagster-snowflake-pyspark library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-snowflake-pyspark library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-snowflake-pyspark-library">
# dagster-snowflake-pyspark library
This library provides an integration with the [Snowflake](https://www.snowflake.com) data
warehouse and PySpark data processing library.
To use this library, you should first ensure that you have an appropriate [Snowflake user](https://docs.snowflake.net/manuals/user-guide/admin-user-management.html) configured to access
your data warehouse.
Related Guides:
- [Using Dagster with Snowflake guide](https://docs.dagster.io/integrations/libraries/snowflake)
- [Snowflake I/O manager reference](https://docs.dagster.io/integrations/libraries/snowflake/reference)
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pyspark.SnowflakePySparkIOManager'>dagster_snowflake_pyspark.SnowflakePySparkIOManager IOManagerDefinition<a href="#dagster_snowflake_pyspark.SnowflakePySparkIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to Snowflake. When
using the SnowflakePySparkIOManager, any inputs and outputs without type annotations will be loaded
as PySpark DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_pyspark import SnowflakePySparkIOManager
from pyspark.sql import DataFrame
from dagster import Definitions, EnvVar
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": SnowflakePySparkIOManager(
database="my_database",
warehouse="my_warehouse", # required for SnowflakePySparkIOManager
account=EnvVar("SNOWFLAKE_ACCOUNT"),
private_key=EnvVar("SNOWFLAKE_PRIVATE_KEY"),
...
)
}
)
```
Note that the warehouse configuration value is required when using the SnowflakePySparkIOManager
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table]
resources={
"io_manager" SnowflakePySparkIOManager(database="my_database", schema="my_schema", ...)
}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: DataFrame) -> DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pyspark.SnowflakePySparkTypeHandler'>`class` dagster_snowflake_pyspark.SnowflakePySparkTypeHandler<a href="#dagster_snowflake_pyspark.SnowflakePySparkTypeHandler" class="hash-link"></a></Link></dt>
<dd>
Plugin for the Snowflake I/O Manager that can store and load PySpark DataFrames as Snowflake tables.
Examples:
```python
from dagster_snowflake import SnowflakeIOManager
from dagster_snowflake_pandas import SnowflakePandasTypeHandler
from dagster_snowflake_pyspark import SnowflakePySparkTypeHandler
from dagster import Definitions, EnvVar
class MySnowflakeIOManager(SnowflakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [SnowflakePandasTypeHandler(), SnowflakePySparkTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": MySnowflakeIOManager(database="MY_DATABASE", account=EnvVar("SNOWFLAKE_ACCOUNT"), warehouse="my_warehouse", ...)
}
)
```
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_snowflake_pyspark.snowflake_pyspark_io_manager'>dagster_snowflake_pyspark.snowflake_pyspark_io_manager IOManagerDefinition<a href="#dagster_snowflake_pyspark.snowflake_pyspark_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
An I/O manager definition that reads inputs from and writes PySpark DataFrames to Snowflake. When
using the snowflake_pyspark_io_manager, any inputs and outputs without type annotations will be loaded
as PySpark DataFrames.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake_pyspark import snowflake_pyspark_io_manager
from pyspark.sql import DataFrame
from dagster import Definitions
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> DataFrame: # the name of the asset will be the table name
...
Definitions(
assets=[my_table],
resources={
"io_manager": snowflake_pyspark_io_manager.configured({
"database": "my_database",
"warehouse": "my_warehouse", # required for snowflake_pyspark_io_manager
"account" : {"env": "SNOWFLAKE_ACCOUNT"},
"private_key": {"env": "SNOWFLAKE_PRIVATE_KEY"},
...
})
}
)
```
Note that the warehouse configuration value is required when using the snowflake_pyspark_io_manager
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table]
resources={"io_manager" snowflake_pyspark_io_manager.configured(
{"database": "my_database", "schema": "my_schema", ...} # will be used as the schema
)}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata “columns” to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: DataFrame) -> DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-snowflake library'
sidebar_position: 1000
title_meta: 'dagster-snowflake library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-snowflake library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-snowflake-library">
# dagster-snowflake library
This library provides an integration with the [Snowflake](https://www.snowflake.com) data
warehouse.
To use this library, you should first ensure that you have an appropriate [Snowflake user](https://docs.snowflake.net/manuals/user-guide/admin-user-management.html) configured to access
your data warehouse.
Related Guides:
- [Using Dagster with Snowflake](https://docs.dagster.io/integrations/libraries/snowflake)
- [Snowflake I/O manager reference](https://docs.dagster.io/integrations/libraries/snowflake/reference)
- [Transitioning data pipelines from development to production](https://docs.dagster.io/guides/operate/dev-to-prod)
- [Testing against production with Dagster+ Branch Deployments](https://docs.dagster.io/deployment/dagster-plus/deploying-code/branch-deployments)
<div class="section" id="i-o-manager">
## I/O manager
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeIOManager'>dagster_snowflake.SnowflakeIOManager IOManagerDefinition<a href="#dagster_snowflake.SnowflakeIOManager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Base class for an IO manager definition that reads inputs from and writes outputs to Snowflake.
Examples:
```python
from dagster_snowflake import SnowflakeIOManager
from dagster_snowflake_pandas import SnowflakePandasTypeHandler
from dagster_snowflake_pyspark import SnowflakePySparkTypeHandler
from dagster import Definitions, EnvVar
class MySnowflakeIOManager(SnowflakeIOManager):
@staticmethod
def type_handlers() -> Sequence[DbTypeHandler]:
return [SnowflakePandasTypeHandler(), SnowflakePySparkTypeHandler()]
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
defs = Definitions(
assets=[my_table],
resources={
"io_manager": MySnowflakeIOManager(database="my_database", account=EnvVar("SNOWFLAKE_ACCOUNT"), ...)
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
defs = Definitions(
assets=[my_table]
resources={
"io_manager" MySnowflakeIOManager(database="my_database", schema="my_schema", ...)
}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata `columns` to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
</div>
<div class="section" id="resources">
## Resources
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeResource'>dagster_snowflake.SnowflakeResource ResourceDefinition<a href="#dagster_snowflake.SnowflakeResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
A resource for connecting to the Snowflake data warehouse.
If connector configuration is not set, SnowflakeResource.get_connection() will return a
[snowflake.connector.Connection](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#object-connection)
object. If connector=”sqlalchemy” configuration is set, then SnowflakeResource.get_connection() will
return a [SQLAlchemy Connection](https://docs.sqlalchemy.org/en/20/core/connections.html#sqlalchemy.engine.Connection)
or a [SQLAlchemy raw connection](https://docs.sqlalchemy.org/en/20/core/connections.html#sqlalchemy.engine.Engine.raw_connection).
A simple example of loading data into Snowflake and subsequently querying that data is shown below:
Examples:
```python
from dagster import job, op
from dagster_snowflake import SnowflakeResource
@op
def get_one(snowflake_resource: SnowflakeResource):
with snowflake_resource.get_connection() as conn:
# conn is a snowflake.connector.Connection object
conn.cursor().execute("SELECT 1")
@job
def my_snowflake_job():
get_one()
my_snowflake_job.execute_in_process(
resources={
'snowflake_resource': SnowflakeResource(
account=EnvVar("SNOWFLAKE_ACCOUNT"),
user=EnvVar("SNOWFLAKE_USER"),
private_key=EnvVar("SNOWFLAKE_PRIVATE_KEY"),
database="MY_DATABASE",
schema="MY_SCHEMA",
warehouse="MY_WAREHOUSE"
)
}
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeConnection'>`class` dagster_snowflake.SnowflakeConnection<a href="#dagster_snowflake.SnowflakeConnection" class="hash-link"></a></Link></dt>
<dd>
A connection to Snowflake that can execute queries. In general this class should not be
directly instantiated, but rather used as a resource in an op or asset via the
[`snowflake_resource()`](#dagster_snowflake.snowflake_resource).
Note that the SnowflakeConnection is only used by the snowflake_resource. The Pythonic SnowflakeResource does
not use this SnowflakeConnection class.
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeConnection.execute_queries'>execute_queries<a href="#dagster_snowflake.SnowflakeConnection.execute_queries" class="hash-link"></a></Link></dt>
<dd>
Execute multiple queries in Snowflake.
Parameters:
- <strong>sql_queries</strong> (<em>str</em>) – List of queries to be executed in series
- <strong>parameters</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Sequence</em><em>[</em><em>Any</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>Any</em><em>, </em><em>Any</em><em>]</em><em>]</em><em>]</em>) – Parameters to be passed to every query. See the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/python-connector-example.html#binding-data) for more information.
- <strong>fetch_results</strong> (<em>bool</em>) – If True, will return the results of the queries as a list. Defaults to False. If True and use_pandas_result is also True, results will be returned as Pandas DataFrames.
- <strong>use_pandas_result</strong> (<em>bool</em>) – If True, will return the results of the queries as a list of a Pandas DataFrames. Defaults to False. If fetch_results is False and use_pandas_result is True, an error will be raised.
Returns: The results of the queries as a list if fetch_results or use_pandas_result is True,
otherwise returns None
Examples:
```python
@op
def create_fresh_database(snowflake: SnowflakeResource):
queries = ["DROP DATABASE IF EXISTS MY_DATABASE", "CREATE DATABASE MY_DATABASE"]
snowflake.execute_queries(
sql_queries=queries
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeConnection.execute_query'>execute_query<a href="#dagster_snowflake.SnowflakeConnection.execute_query" class="hash-link"></a></Link></dt>
<dd>
Execute a query in Snowflake.
Parameters:
- <strong>sql</strong> (<em>str</em>) – the query to be executed
- <strong>parameters</strong> (<em>Optional</em><em>[</em><em>Union</em><em>[</em><em>Sequence</em><em>[</em><em>Any</em><em>]</em><em>, </em><em>Mapping</em><em>[</em><em>Any</em><em>, </em><em>Any</em><em>]</em><em>]</em><em>]</em>) – Parameters to be passed to the query. See the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/python-connector-example.html#binding-data) for more information.
- <strong>fetch_results</strong> (<em>bool</em>) – If True, will return the result of the query. Defaults to False. If True and use_pandas_result is also True, results will be returned as a Pandas DataFrame.
- <strong>use_pandas_result</strong> (<em>bool</em>) – If True, will return the result of the query as a Pandas DataFrame. Defaults to False. If fetch_results is False and use_pandas_result is True, an error will be raised.
Returns: The result of the query if fetch_results or use_pandas_result is True, otherwise returns None
Examples:
```python
@op
def drop_database(snowflake: SnowflakeResource):
snowflake.execute_query(
"DROP DATABASE IF EXISTS MY_DATABASE"
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeConnection.get_connection'>get_connection<a href="#dagster_snowflake.SnowflakeConnection.get_connection" class="hash-link"></a></Link></dt>
<dd>
Gets a connection to Snowflake as a context manager.
If using the execute_query, execute_queries, or load_table_from_local_parquet methods,
you do not need to create a connection using this context manager.
Parameters: <strong>raw_conn</strong> (<em>bool</em>) – If using the sqlalchemy connector, you can set raw_conn to True to create a raw
connection. Defaults to True.
Examples:
```python
@op(
required_resource_keys={"snowflake"}
)
def get_query_status(query_id):
with context.resources.snowflake.get_connection() as conn:
# conn is a Snowflake Connection object or a SQLAlchemy Connection if
# sqlalchemy is specified as the connector in the Snowflake Resource config
return conn.get_query_status(query_id)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake.SnowflakeConnection.load_table_from_local_parquet'>load_table_from_local_parquet<a href="#dagster_snowflake.SnowflakeConnection.load_table_from_local_parquet" class="hash-link"></a></Link></dt>
<dd>
Stores the content of a parquet file to a Snowflake table.
Parameters:
- <strong>src</strong> (<em>str</em>) – the name of the file to store in Snowflake
- <strong>table</strong> (<em>str</em>) – the name of the table to store the data. If the table does not exist, it will be created. Otherwise the contents of the table will be replaced with the data in src
Examples:
```python
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
@op
def write_parquet_file(snowflake: SnowflakeResource):
df = pd.DataFrame({"one": [1, 2, 3], "ten": [11, 12, 13]})
table = pa.Table.from_pandas(df)
pq.write_table(table, "example.parquet')
snowflake.load_table_from_local_parquet(
src="example.parquet",
table="MY_TABLE"
)
```
</dd>
</dl>
</dd>
</dl>
</div>
<div class="section" id="data-freshness">
## Data freshness
<dl>
<dt><Link class="anchor" id='dagster_snowflake.fetch_last_updated_timestamps'>dagster_snowflake.fetch_last_updated_timestamps<a href="#dagster_snowflake.fetch_last_updated_timestamps" class="hash-link"></a></Link></dt>
<dd>
Fetch the last updated times of a list of tables in Snowflake.
If the underlying query to fetch the last updated time returns no results, a ValueError will be raised.
Parameters:
- <strong>snowflake_connection</strong> (<em>Union</em><em>[</em><em>SqlDbConnection</em><em>, </em>[*SnowflakeConnection*](#dagster_snowflake.SnowflakeConnection)<em>]</em>) – A connection to Snowflake. Accepts either a SnowflakeConnection or a sqlalchemy connection object, which are the two types of connections emittable from the snowflake resource.
- <strong>schema</strong> (<em>str</em>) – The schema of the tables to fetch the last updated time for.
- <strong>tables</strong> (<em>Sequence</em><em>[</em><em>str</em><em>]</em>) – A list of table names to fetch the last updated time for.
- <strong>database</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The database of the table. Only required if the connection has not been set with a database.
- <strong>ignore_missing_tables</strong> (<em>Optional</em><em>[</em><em>bool</em><em>]</em>) – If True, tables not found in Snowflake will be excluded from the result.
Returns: A dictionary of table names to their last updated time in UTC.Return type: Mapping[str, datetime]
</dd>
</dl>
</div>
<div class="section" id="ops">
## Ops
<dl>
<dt><Link class="anchor" id='dagster_snowflake.snowflake_op_for_query'>dagster_snowflake.snowflake_op_for_query<a href="#dagster_snowflake.snowflake_op_for_query" class="hash-link"></a></Link></dt>
<dd>
This function is an op factory that constructs an op to execute a snowflake query.
Note that you can only use <cite>snowflake_op_for_query</cite> if you know the query you’d like to
execute at graph construction time. If you’d like to execute queries dynamically during
job execution, you should manually execute those queries in your custom op using the
snowflake resource.
Parameters:
- <strong>sql</strong> (<em>str</em>) – The sql query that will execute against the provided snowflake resource.
- <strong>parameters</strong> (<em>dict</em>) – The parameters for the sql query.
Returns: Returns the constructed op definition.Return type: [OpDefinition](../../../api/dagster/ops.mdx#dagster.OpDefinition)
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_snowflake.build_snowflake_io_manager'>dagster_snowflake.build_snowflake_io_manager IOManagerDefinition<a href="#dagster_snowflake.build_snowflake_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Builds an IO manager definition that reads inputs from and writes outputs to Snowflake.
Parameters:
- <strong>type_handlers</strong> (<em>Sequence</em><em>[</em><em>DbTypeHandler</em><em>]</em>) – Each handler defines how to translate between slices of Snowflake tables and an in-memory type - e.g. a Pandas DataFrame. If only one DbTypeHandler is provided, it will be used as the default_load_type.
- <strong>default_load_type</strong> (<em>Type</em>) – When an input has no type annotation, load it as this type.
Returns: IOManagerDefinition
Examples:
```python
from dagster_snowflake import build_snowflake_io_manager
from dagster_snowflake_pandas import SnowflakePandasTypeHandler
from dagster_snowflake_pyspark import SnowflakePySparkTypeHandler
from dagster import Definitions
@asset(
key_prefix=["my_prefix"]
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame: # the name of the asset will be the table name
...
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_second_table() -> pd.DataFrame: # the name of the asset will be the table name
...
snowflake_io_manager = build_snowflake_io_manager([SnowflakePandasTypeHandler(), SnowflakePySparkTypeHandler()])
Definitions(
assets=[my_table, my_second_table],
resources={
"io_manager": snowflake_io_manager.configured({
"database": "my_database",
"account" : {"env": "SNOWFLAKE_ACCOUNT"}
...
})
}
)
```
You can set a default schema to store the assets using the `schema` configuration value of the Snowflake I/O
Manager. This schema will be used if no other schema is specified directly on an asset or op.
```python
Definitions(
assets=[my_table]
resources={"io_manager" snowflake_io_manager.configured(
{"database": "my_database", "schema": "my_schema", ...} # will be used as the schema
)}
)
```
On individual assets, you an also specify the schema where they should be stored using metadata or
by adding a `key_prefix` to the asset key. If both `key_prefix` and metadata are defined, the metadata will
take precedence.
```python
@asset(
key_prefix=["my_schema"] # will be used as the schema in snowflake
)
def my_table() -> pd.DataFrame:
...
@asset(
metadata={"schema": "my_schema"} # will be used as the schema in snowflake
)
def my_other_table() -> pd.DataFrame:
...
```
For ops, the schema can be specified by including a “schema” entry in output metadata.
```python
@op(
out={"my_table": Out(metadata={"schema": "my_schema"})}
)
def make_my_table() -> pd.DataFrame:
...
```
If none of these is provided, the schema will default to “public”.
To only use specific columns of a table as input to a downstream op or asset, add the metadata `columns` to the
In or AssetIn.
```python
@asset(
ins={"my_table": AssetIn("my_table", metadata={"columns": ["a"]})}
)
def my_table_a(my_table: pd.DataFrame) -> pd.DataFrame:
# my_table will just contain the data from column "a"
...
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_snowflake.snowflake_resource'>dagster_snowflake.snowflake_resource ResourceDefinition<a href="#dagster_snowflake.snowflake_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
A resource for connecting to the Snowflake data warehouse. The returned resource object is an
instance of [`SnowflakeConnection`](#dagster_snowflake.SnowflakeConnection).
A simple example of loading data into Snowflake and subsequently querying that data is shown below:
Examples:
```python
from dagster import job, op
from dagster_snowflake import snowflake_resource
@op(required_resource_keys={'snowflake'})
def get_one(context):
context.resources.snowflake.execute_query('SELECT 1')
@job(resource_defs={'snowflake': snowflake_resource})
def my_snowflake_job():
get_one()
my_snowflake_job.execute_in_process(
run_config={
'resources': {
'snowflake': {
'config': {
'account': {'env': 'SNOWFLAKE_ACCOUNT'},
'user': {'env': 'SNOWFLAKE_USER'},
'private_key': {'env': 'SNOWFLAKE_PRIVATE_KEY'},
'database': {'env': 'SNOWFLAKE_DATABASE'},
'schema': {'env': 'SNOWFLAKE_SCHEMA'},
'warehouse': {'env': 'SNOWFLAKE_WAREHOUSE'},
}
}
}
}
)
```
</dd>
</dl>
</div></div>
---
---
title: 'dagster-spark library'
sidebar_position: 1000
title_meta: 'dagster-spark library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-spark library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-spark-library">
# dagster-spark library
<dl>
<dt><Link class="anchor" id='dagster_spark.SparkOpError'>`class` dagster_spark.SparkOpError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/types.py#L10' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.SparkOpError" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_spark.define_spark_config'>dagster_spark.define_spark_config <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/configs.py#L15' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.define_spark_config" class="hash-link"></a></Link></dt>
<dd>
Spark configuration.
See the Spark documentation for reference:
[https://spark.apache.org/docs/latest/submitting-applications.html](https://spark.apache.org/docs/latest/submitting-applications.html)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_spark.create_spark_op'>dagster_spark.create_spark_op <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/ops.py#L13' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.create_spark_op" class="hash-link"></a></Link></dt>
<dd>
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_spark.construct_spark_shell_command'>dagster_spark.construct_spark_shell_command <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/utils.py#L48' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.construct_spark_shell_command" class="hash-link"></a></Link></dt>
<dd>
Constructs the spark-submit command for a Spark job.
</dd>
</dl>
<div class="section" id="spark-declarative-pipelines">
## Spark declarative pipelines
<dl>
<dt><Link class="anchor" id='dagster_spark.SparkDeclarativePipelineComponent'>`class` dagster_spark.SparkDeclarativePipelineComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/component.py#L54' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.SparkDeclarativePipelineComponent" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
State-backed component for Spark Declarative Pipelines (SDP).
Discovers datasets via spark-pipelines dry-run (or source_only), caches state,
and builds a multi_asset that runs spark-pipelines run and yields MaterializeResults.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_spark.components.spark_declarative_pipeline.SparkPipelinesResource'>`class` dagster_spark.components.spark_declarative_pipeline.SparkPipelinesResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/resource.py#L27' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.components.spark_declarative_pipeline.SparkPipelinesResource" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Dagster resource for Spark Declarative Pipelines: discovery and run.
Use discover_datasets to get datasets from spark-pipelines dry-run (or source_only).
Use run_and_observe inside an asset to run the pipeline and yield MaterializeResults.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_spark.components.spark_declarative_pipeline.SparkDeclarativePipelineScaffolder'>`class` dagster_spark.components.spark_declarative_pipeline.SparkDeclarativePipelineScaffolder <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/scaffolder.py#L9' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.components.spark_declarative_pipeline.SparkDeclarativePipelineScaffolder" class="hash-link"></a></Link></dt>
<dd>
:::info preview
This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.
:::
Scaffolds a Spark Declarative Pipeline component defs.yaml and pipeline spec path.
</dd>
</dl>
</div>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_spark.spark_resource'>dagster_spark.spark_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-spark/dagster_spark/resources.py#L67' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_spark.spark_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
</div></div>
---
---
title: 'dagster-ssh library'
sidebar_position: 1000
title_meta: 'dagster-ssh library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-ssh library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-ssh-library">
# dagster-ssh library
This library provides an integration with SSH and SFTP.
<dl>
<dt><Link class="anchor" id='dagster_ssh.SSHResource'>`class` dagster_ssh.SSHResource <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-ssh/dagster_ssh/resources.py#L47' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_ssh.SSHResource" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A Dagster resource for establishing SSH connections and performing remote file operations.
This resource leverages the Paramiko library to provide robust SSH connectivity,
including support for key-based and password authentication, tunneling, and SFTP transfers.
Parameters:
- <strong>remote_host</strong> (<em>str</em>) – The hostname or IP address of the remote server to connect to.
- <strong>remote_port</strong> (<em>Optional</em><em>[</em><em>int</em><em>]</em>) – The SSH port on the remote host. Defaults to standard SSH port 22.
- <strong>username</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The username for SSH authentication. If not provided, defaults to the current system user.
- <strong>password</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – The password for SSH authentication. Not recommended for production use; prefer key-based authentication.
- <strong>key_file</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – Path to the SSH private key file for authentication.
- <strong>key_string</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em>) – SSH private key as a string for authentication.
- <strong>timeout</strong> (<em>int</em><em>, </em><em>optional</em>) – Connection timeout in seconds. Defaults to 10.
- <strong>keepalive_interval</strong> (<em>int</em><em>, </em><em>optional</em>) – Interval for sending SSH keepalive packets. (Defaults to 30 seconds.)
- <strong>compress</strong> (<em>bool</em><em>, </em><em>optional</em>) – Whether to compress the SSH transport stream. Defaults to True.
- <strong>no_host_key_check</strong> (<em>bool</em><em>, </em><em>optional</em>) – Disable host key verification.
- <strong>allow_host_key_change</strong> (<em>bool</em><em>, </em><em>optional</em>) – Allow connections to hosts with changed host keys. (Defaults to False.)
Example:
Creating an SSH resource with key-based authentication:
```python
ssh_resource = SSHResource(
remote_host="example.com",
username="myuser", key_file="/path/to/private/key"
)
```
Creating an SSH resource with password authentication:
```python
ssh_resource = SSHResource(
remote_host="example.com",
username="myuser",
password="my_secure_password"
)
```
Using the resource to transfer a file:
```python
local_file = ssh_resource.sftp_get("/remote/path/file.txt", "/local/path/file.txt")
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_ssh.ssh_resource'>dagster_ssh.ssh_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-ssh/dagster_ssh/resources.py#L294' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_ssh.ssh_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
A Dagster resource factory for creating SSHResource instances.
This function converts Dagster resource context configuration into an SSHResource
that can be used for remote SSH connections and file operations.
Parameters: <strong>init_context</strong> ([*InitResourceContext*](../../../api/dagster/resources.mdx#dagster.InitResourceContext)) – The Dagster resource initialization context containing configuration parameters.Returns: A configured SSH resource ready for use in Dagster pipelines.Return type: [SSHResource](#dagster_ssh.SSHResource)
Example:
Configuring the SSH resource in a Dagster pipeline:
```python
from dagster import Definitions, job, op
from dagster_ssh import ssh_resource
@op
def transfer_files(ssh):
ssh.sftp_get("/remote/file", "/local/file")
@job
def my_ssh_job():
transfer_files(ssh=ssh_resource.configured({
"remote_host": "example.com",
"username": "myuser",
"key_file": "/path/to/private/key"
}))
Definitions(jobs=[my_ssh_job])
```
</dd>
</dl>
</div>
---
---
title: 'dagster-tableau library'
sidebar_position: 1000
title_meta: 'dagster-tableau library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-tableau library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-tableau-library">
# dagster-tableau library
Dagster allows you to represent your Tableau workspace as assets, alongside other your other
technologies like dbt and Sling. This allows you to see how your Tableau assets are connected to
your other data assets, and how changes to other data assets might impact your Tableau workspace.
<div class="section" id="component">
## Component
<dl>
<dt><Link class="anchor" id='dagster_tableau.TableauComponent'>`class` dagster_tableau.TableauComponent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/components/tableau_component.py#L240' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.TableauComponent" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Pulls in the contents of a Tableau workspace into Dagster assets.
Example:
```yaml
# defs.yaml
type: dagster_tableau.TableauComponent
attributes:
workspace:
type: cloud
connected_app_client_id: "{{ env.TABLEAU_CLIENT_ID }}"
connected_app_secret_id: "{{ env.TABLEAU_SECRET_ID }}"
connected_app_secret_value: "{{ env.TABLEAU_SECRET_VALUE }}"
username: "{{ env.TABLEAU_USERNAME }}"
site_name: my_site
pod_name: 10ax
tableau_selector:
workbooks:
ids:
- "abc123-def456"
- "xyz789-uvw012"
projects:
names:
- "my_project"
```
<dl>
<dt><Link class="anchor" id='dagster_tableau.TableauComponent.get_asset_spec'>get_asset_spec <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/components/tableau_component.py#L334' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.TableauComponent.get_asset_spec" class="hash-link"></a></Link></dt>
<dd>
Generates an AssetSpec for a given Tableau content item.
This method can be overridden in a subclass to customize how Tableau content
(workbooks, dashboards, sheets, data sources) are converted to Dagster asset specs.
By default, it delegates to the configured DagsterTableauTranslator.
Parameters: <strong>data</strong> – The TableauTranslatorData containing information about the Tableau content
item and workspaceReturns: An AssetSpec that represents the Tableau content as a Dagster asset
Example:
Override this method to add custom metadata based on content properties:
```python
from dagster_tableau import TableauComponent
from dagster import AssetSpec
class CustomTableauComponent(TableauComponent):
def get_asset_spec(self, data):
base_spec = super().get_asset_spec(data)
return base_spec.replace_attributes(
metadata={
**base_spec.metadata,
"tableau_type": data.content_data.content_type,
"project": data.content_data.properties.get("project", {}).get("name")
}
)
```
</dd>
</dl>
</dd>
</dl>
To use the Tableau component, see the [Tableau component integration guide](https://docs.dagster.io/integrations/libraries/tableau).
<div class="section" id="yaml-configuration">
### YAML configuration
When you scaffold a Tableau component definition, the following `defs.yaml` configuration file will be created:
```yaml
type: dagster_tableau.TableauComponent
attributes:
workspace:
type: cloud
connected_app_client_id: "{{ env.TABLEAU_CONNECTED_APP_CLIENT_ID }}"
connected_app_secret_id: "{{ env.TABLEAU_CONNECTED_APP_SECRET_ID }}"
connected_app_secret_value: "{{ env.TABLEAU_CONNECTED_APP_SECRET_VALUE }}"
username: "{{ env.TABLEAU_USERNAME }}"
site_name: "{{ env.TABLEAU_SITE_NAME }}"
pod_name: "{{ env.TABLEAU_POD_NAME }}"
```
</div></div>
<div class="section" id="tableau-api">
## Tableau API
Here, we provide interfaces to manage Tableau projects using the Tableau API.
<div class="section" id="assets-tableau-api">
### Assets (Tableau API)
<dl>
<dt><Link class="anchor" id='dagster_tableau.TableauCloudWorkspace'>`class` dagster_tableau.TableauCloudWorkspace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/resources.py#L993' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.TableauCloudWorkspace" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Represents a workspace in Tableau Cloud and provides utilities
to interact with Tableau APIs.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.TableauServerWorkspace'>`class` dagster_tableau.TableauServerWorkspace <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/resources.py#L1012' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.TableauServerWorkspace" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Represents a workspace in Tableau Server and provides utilities
to interact with Tableau APIs.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.DagsterTableauTranslator'>`class` dagster_tableau.DagsterTableauTranslator <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/translator.py#L217' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.DagsterTableauTranslator" class="hash-link"></a></Link></dt>
<dd>
Translator class which converts raw response data from the Tableau API into AssetSpecs.
Subclass this class to implement custom logic for each type of Tableau content.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.tableau_assets'>@dagster_tableau.tableau_assets <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/asset_decorator.py#L13' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.tableau_assets" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Create a definition for how to refresh the extracted data sources and views of a given Tableau workspace.
Parameters:
- <strong>workspace</strong> (<em>Union</em><em>[</em>[*TableauCloudWorkspace*](#dagster_tableau.TableauCloudWorkspace)<em>, </em>[*TableauServerWorkspace*](#dagster_tableau.TableauServerWorkspace)<em>]</em>) – The Tableau workspace to fetch assets from.
- <strong>name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the op.
- <strong>group_name</strong> (<em>Optional</em><em>[</em><em>str</em><em>]</em><em>, </em><em>optional</em>) – The name of the asset group.
- <strong>dagster_tableau_translator</strong> (<em>Optional</em><em>[</em>[*DagsterTableauTranslator*](#dagster_tableau.DagsterTableauTranslator)<em>]</em><em>, </em><em>optional</em>) – The translator to use to convert Tableau content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterTableauTranslator`](#dagster_tableau.DagsterTableauTranslator).
- <strong>workbook_selector_fn</strong> (<em>Optional</em><em>[</em><em>WorkbookSelectorFn</em><em>]</em>) – <span className="flag flag-info">beta</span> A function that allows for filtering which Tableau workbook assets are created for, including data sources, sheets and dashboards.
Examples:
Refresh extracted data sources and views in Tableau:
```python
from dagster_tableau import TableauCloudWorkspace, tableau_assets
import dagster as dg
tableau_workspace = TableauCloudWorkspace(
connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"),
connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"),
connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"),
username=dg.EnvVar("TABLEAU_USERNAME"),
site_name=dg.EnvVar("TABLEAU_SITE_NAME"),
pod_name=dg.EnvVar("TABLEAU_POD_NAME"),
)
@tableau_assets(
workspace=tableau_workspace,
name="tableau_workspace_assets",
group_name="tableau",
)
def tableau_workspace_assets(context: dg.AssetExecutionContext, tableau: TableauCloudWorkspace):
yield from tableau.refresh_and_poll(context=context)
defs = dg.Definitions(
assets=[tableau_workspace_assets],
resources={"tableau": tableau_workspace},
)
```
Refresh extracted data sources and views in Tableau with a custom translator:
```python
from dagster_tableau import (
DagsterTableauTranslator,
TableauTranslatorData,
TableauCloudWorkspace,
tableau_assets
)
import dagster as dg
class CustomDagsterTableauTranslator(DagsterTableauTranslator):
def get_asset_spec(self, data: TableauTranslatorData) -> dg.AssetSpec:
default_spec = super().get_asset_spec(data)
return default_spec.replace_attributes(
key=default_spec.key.with_prefix("my_prefix"),
)
tableau_workspace = TableauCloudWorkspace(
connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"),
connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"),
connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"),
username=dg.EnvVar("TABLEAU_USERNAME"),
site_name=dg.EnvVar("TABLEAU_SITE_NAME"),
pod_name=dg.EnvVar("TABLEAU_POD_NAME"),
)
@tableau_assets(
workspace=tableau_workspace,
name="tableau_workspace_assets",
group_name="tableau",
dagster_tableau_translator=CustomDagsterTableauTranslator(),
)
def tableau_workspace_assets(context: dg.AssetExecutionContext, tableau: TableauCloudWorkspace):
yield from tableau.refresh_and_poll(context=context)
defs = dg.Definitions(
assets=[tableau_workspace_assets],
resources={"tableau": tableau_workspace},
)
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.load_tableau_asset_specs'>dagster_tableau.load_tableau_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/resources.py#L966' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.load_tableau_asset_specs" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
Returns a list of AssetSpecs representing the Tableau content in the workspace.
Parameters:
- <strong>workspace</strong> (<em>Union</em><em>[</em>[*TableauCloudWorkspace*](#dagster_tableau.TableauCloudWorkspace)<em>, </em>[*TableauServerWorkspace*](#dagster_tableau.TableauServerWorkspace)<em>]</em>) – The Tableau workspace to fetch assets from.
- <strong>dagster_tableau_translator</strong> (<em>Optional</em><em>[</em>[*DagsterTableauTranslator*](#dagster_tableau.DagsterTableauTranslator)<em>]</em>) – The translator to use to convert Tableau content into [`dagster.AssetSpec`](../../../api/dagster/assets.mdx#dagster.AssetSpec). Defaults to [`DagsterTableauTranslator`](#dagster_tableau.DagsterTableauTranslator).
- <strong>workbook_selector_fn</strong> (<em>Optional</em><em>[</em><em>WorkbookSelectorFn</em><em>]</em>) – <span className="flag flag-info">beta</span> A function that allows for filtering which Tableau workbook assets are created for, including data sources, sheets and dashboards.
Returns: The set of assets representing the Tableau content in the workspace.Return type: List[[AssetSpec](../../../api/dagster/assets.mdx#dagster.AssetSpec)]
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.build_tableau_materializable_assets_definition'>dagster_tableau.build_tableau_materializable_assets_definition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/assets.py#L12' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.build_tableau_materializable_assets_definition" class="hash-link"></a></Link></dt>
<dd>
:::info beta
This API is currently in beta, and may have breaking changes in minor version releases, with behavior changes in patch releases.
:::
:::warning superseded
This API has been superseded.
Use `tableau_assets` decorator instead..
:::
Returns the AssetsDefinition of the materializable assets in the Tableau workspace.
Parameters:
- <strong>resource_key</strong> (<em>str</em>) – The resource key to use for the Tableau resource.
- <strong>specs</strong> (<em>Sequence</em><em>[</em>[*AssetSpec*](../../../api/dagster/assets.mdx#dagster.AssetSpec)<em>]</em>) – The asset specs of the executable assets in the Tableau workspace.
- <strong>refreshable_workbook_ids</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) –
<span className="flag flag-warning">deprecated</span> A list of workbook IDs. The provided workbooks must have extracts as data sources and be refreshable in Tableau.
When materializing your Tableau assets, the workbooks provided are refreshed, refreshing their sheets and dashboards before pulling their data in Dagster.
- <strong>refreshable_data_source_ids</strong> (<em>Optional</em><em>[</em><em>Sequence</em><em>[</em><em>str</em><em>]</em><em>]</em>) –
A list of data source IDs. The provided data sources must have extracts and be refreshable in Tableau.
When materializing your Tableau assets, the provided data source are refreshed, refreshing upstream sheets and dashboards before pulling their data in Dagster.
Returns: The AssetsDefinition of the executable assets in the Tableau workspace.Return type: [AssetsDefinition](../../../api/dagster/assets.mdx#dagster.AssetsDefinition)
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_tableau.parse_tableau_external_and_materializable_asset_specs'>dagster_tableau.parse_tableau_external_and_materializable_asset_specs <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-tableau/dagster_tableau/asset_utils.py#L40' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_tableau.parse_tableau_external_and_materializable_asset_specs" class="hash-link"></a></Link></dt>
<dd>
Parses a list of Tableau AssetSpecs provided as input and return two lists of AssetSpecs,
one for the Tableau external assets and another one for the Tableau materializable assets.
In Tableau, data sources are considered external assets,
while sheets and dashboards are considered materializable assets.
Parameters:
- <strong>specs</strong> (<em>Sequence</em><em>[</em>[*AssetSpec*](../../../api/dagster/assets.mdx#dagster.AssetSpec)<em>]</em>) – The asset specs of the assets in the Tableau workspace.
- <strong>include_data_sources_with_extracts</strong> (<em>bool</em>) – Whether to include published data sources with extracts in materializable assets.
Returns:
A named tuple representing the parsed Tableau asset specs
as <cite>external_asset_specs</cite> and <cite>materializable_asset_specs</cite>.
Return type: ParsedTableauAssetSpecs
</dd>
</dl>
</div></div></div>
---
---
title: 'dagster-twilio library'
sidebar_position: 1000
title_meta: 'dagster-twilio library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-twilio library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-twilio-library">
# dagster-twilio library
This library provides an integration with Twilio.
<dl>
<dt><Link class="anchor" id='dagster_twilio.TwilioResource'>dagster_twilio.TwilioResource ResourceDefinition<a href="#dagster_twilio.TwilioResource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
This resource is for connecting to Twilio.
</dd>
</dl>
<div class="section" id="legacy">
## Legacy
<dl>
<dt><Link class="anchor" id='dagster_twilio.twilio_resource'>dagster_twilio.twilio_resource ResourceDefinition<a href="#dagster_twilio.twilio_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
</dd>
</dl>
</div></div>
---
---
title: 'dagster-wandb library'
sidebar_position: 1000
title_meta: 'dagster-wandb library API Documentation - Build Better Data Pipelines | Python Reference Documentation for Dagster'
description: 'dagster-wandb library Dagster API | Comprehensive Python API documentation for Dagster, the data orchestration platform. Learn how to build, test, and maintain data pipelines with our detailed guides and examples.'
last_update:
date: '2026-09-16'
custom_edit_url: null
---
<div class="section" id="dagster-wandb-library">
# dagster-wandb library
This library provides a Dagster integration with [Weights & Biases](https://wandb.ai).
Use Dagster and Weights & Biases (W&B) to orchestrate your MLOps pipelines and maintain ML assets.
The integration with W&B makes it easy within Dagster to:
- use and create [W&B Artifacts](https://docs.wandb.ai/guides/artifacts).
- use and create Registered Models in the [W&B Model Registry](https://docs.wandb.ai/guides/models).
- run training jobs on dedicated compute using [W&B Launch](https://docs.wandb.ai/guides/launch).
- use the [wandb](https://github.com/wandb/wandb) client in ops and assets.
<div class="section" id="useful-links">
## Useful links
For a complete set of documentation, see [Dagster integration](https://docs.wandb.ai/guides/integrations/dagster) on the W&B website.
For full-code examples, see [examples/with_wandb](https://github.com/dagster-io/dagster/tree/master/examples/with_wandb) in the Dagster’s Github repo.
</div>
<div class="section" id="resource">
## Resource
<dl>
<dt><Link class="anchor" id='dagster_wandb.wandb_resource'>dagster_wandb.wandb_resource ResourceDefinition <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/resources.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.wandb_resource" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Dagster resource used to communicate with the W&B API. It’s useful when you want to use the
wandb client within your ops and assets. It’s a required resources if you are using the W&B IO
Manager.
It automatically authenticates using the provided API key.
For a complete set of documentation, see [Dagster integration](https://docs.wandb.ai/guides/integrations/dagster).
To configure this resource, we recommend using the [configured](https://legacy-docs.dagster.io/concepts/configuration/configured) method.
<strong>Example:</strong>
```python
from dagster import job
from dagster_wandb import wandb_resource
my_wandb_resource = wandb_resource.configured({"api_key": {"env": "WANDB_API_KEY"}})
@job(resource_defs={"wandb_resource": my_wandb_resource})
def my_wandb_job():
...
```
</dd>
</dl>
</div>
<div class="section" id="i-o-manager">
## I/O Manager
<dl>
<dt><Link class="anchor" id='dagster_wandb.wandb_artifacts_io_manager'>dagster_wandb.wandb_artifacts_io_manager IOManager <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/io_manager.py#L576' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.wandb_artifacts_io_manager" class="hash-link"></a></Link></dt>
<dd>
<div className='lineblock'> </div>
Dagster IO Manager to create and consume W&B Artifacts.
It allows any Dagster @op or @asset to create and consume W&B Artifacts natively.
For a complete set of documentation, see [Dagster integration](https://docs.wandb.ai/guides/integrations/dagster).
<strong>Example:</strong>
```python
@repository
def my_repository():
return [
*with_resources(
load_assets_from_current_module(),
resource_defs={
"wandb_config": make_values_resource(
entity=str,
project=str,
),
"wandb_resource": wandb_resource.configured(
{"api_key": {"env": "WANDB_API_KEY"}}
),
"wandb_artifacts_manager": wandb_artifacts_io_manager.configured(
{"cache_duration_in_minutes": 60} # only cache files for one hour
),
},
resource_config_by_key={
"wandb_config": {
"config": {
"entity": "my_entity",
"project": "my_project"
}
}
},
),
]
@asset(
name="my_artifact",
metadata={
"wandb_artifact_configuration": {
"type": "dataset",
}
},
io_manager_key="wandb_artifacts_manager",
)
def create_dataset():
return [1, 2, 3]
```
</dd>
</dl>
<div class="section" id="config">
### Config
<dl>
<dt><Link class="anchor" id='dagster_wandb.WandbArtifactConfiguration'>`class` dagster_wandb.WandbArtifactConfiguration <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/types.py#L11' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.WandbArtifactConfiguration" class="hash-link"></a></Link></dt>
<dd>
W&B Artifacts IO Manager configuration. Useful for type checking.
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_wandb.SerializationModule'>`class` dagster_wandb.SerializationModule <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/types.py#L4' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.SerializationModule" class="hash-link"></a></Link></dt>
<dd>
W&B Artifacts IO Manager configuration of the serialization module. Useful for type checking.
</dd>
</dl>
</div>
<div class="section" id="errors">
### Errors
<dl>
<dt><Link class="anchor" id='dagster_wandb.WandbArtifactsIOManagerError'>`exception` dagster_wandb.WandbArtifactsIOManagerError <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/utils/errors.py#L1' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.WandbArtifactsIOManagerError" class="hash-link"></a></Link></dt>
<dd>
Represents an execution error of the W&B Artifacts IO Manager.
</dd>
</dl>
</div></div>
<div class="section" id="ops">
## Ops
<dl>
<dt><Link class="anchor" id='dagster_wandb.run_launch_agent'>dagster_wandb.run_launch_agent <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/launch/ops.py#L23' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.run_launch_agent" class="hash-link"></a></Link></dt>
<dd>
It starts a Launch Agent and runs it as a long running process until stopped manually.
Agents are processes that poll launch queues and execute the jobs (or dispatch them to external
services to be executed) in order.
<strong>Example:</strong>
```YAML
# config.yaml
resources:
wandb_config:
config:
entity: my_entity
project: my_project
ops:
run_launch_agent:
config:
max_jobs: -1
queues:
- my_dagster_queue
```
```python
from dagster_wandb.launch.ops import run_launch_agent
from dagster_wandb.resources import wandb_resource
from dagster import job, make_values_resource
@job(
resource_defs={
"wandb_config": make_values_resource(
entity=str,
project=str,
),
"wandb_resource": wandb_resource.configured(
{"api_key": {"env": "WANDB_API_KEY"}}
),
},
)
def run_launch_agent_example():
run_launch_agent()
```
</dd>
</dl>
<dl>
<dt><Link class="anchor" id='dagster_wandb.run_launch_job'>dagster_wandb.run_launch_job <a href='https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-wandb/dagster_wandb/launch/ops.py#L85' className='source-link' target='_blank' rel='noopener noreferrer'>[source]</a><a href="#dagster_wandb.run_launch_job" class="hash-link"></a></Link></dt>
<dd>
Executes a Launch job.
A Launch job is assigned to a queue in order to be executed. You can create a queue or use the
default one. Make sure you have an active agent listening to that queue. You can run an agent
inside your Dagster instance but can also consider using a deployable agent in Kubernetes.
<strong>Example:</strong>
```YAML
# config.yaml
resources:
wandb_config:
config:
entity: my_entity
project: my_project
ops:
my_launched_job:
config:
entry_point:
- python
- train.py
queue: my_dagster_queue
uri: https://github.com/wandb/example-dagster-integration-with-launch
```
```python
from dagster_wandb.launch.ops import run_launch_job
from dagster_wandb.resources import wandb_resource
from dagster import job, make_values_resource
@job(
resource_defs={
"wandb_config": make_values_resource(
entity=str,
project=str,
),
"wandb_resource": wandb_resource.configured(
{"api_key": {"env": "WANDB_API_KEY"}}
),
},
)
def run_launch_job_example():
run_launch_job.alias("my_launched_job")() # we rename the job with an alias
```
</dd>
</dl>
</div></div>