Alert policies define which events will trigger an alert, the conditions under which an alert will be sent, and how the alert will be sent.
Asset based alert policies can trigger on asset materialization failure or success, as well as asset check error, warn, passed, or failure to execute. An asset group or asset key can be provided to asset based alert policies, which limits notifications to only fire if the asset group or asset key matches the materialized asset. In the case of checks, notifications will only be sent if the asset group/key to which the check is attached matches. Note: Asset based alert policies are still experimental, and may be subject to change as we gather user feedback.
Job run based alert policies include a set of configured tags. If an alert policy has no configured tags, all jobs will be eligible for that alert. Otherwise, only jobs that contain all the tags for a given alert policy are eligible for that alert.
Alert policies created for schedule/sensor tick failure will apply to all schedules/sensors. However, you will only receive alerts when the schedule/sensor changes from a state of succeeding to failing, so subsequent failures will not trigger new alerts.
Code location error alert policies will trigger when a code location fails to load due to an error.
Agent downtime alert policies will trigger when a Hybrid agent hasn't heartbeated within the last 5 minutes.
Alert policies are configured on a per-deployment basis. For example, asset alerts configured in a prod deployment are only applicable to assets in that deployment.
Currently, Slack and email notifications are supported.
From the Alert Policy type drop-down, select the type of alert to create.
In the Create alert policy window, fill in the following:
Alert policy name - Enter a name for the alert policy. For example, slack_urgent_failure
Description - Enter a description for the alert policy
For asset-based alerts, fill out these additional options:
Asset group - Select the asset group to monitor. You will have the option to select from all asset groups in the deployment.
Asset key - Select the asset key to monitor. You will have the option to select from all asset keys in the deployment. Note: If you select an asset group, you will not be able to select an asset key.
Events - Select whether the alert should trigger on asset materialization failure, asset materialization success, asset check error, asset check warn, asset check passed, or asset check failure to execute
For job-based alerts, fill out these additional options:
Tags - Add tag(s) for the alert policy. Jobs with these tags will trigger the alert. For example: level:critical or team:sales
Events - Select whether the alert should trigger on job success, failure, or both
Notification service - Select the service for the alert policy:
Slack - If you haven't connected Slack, click the Connect to Slack button to add the Dagster Cloud Slack app to your workspace. After the installation completes, invite the @Dagster Cloud bot user to the desired channel.
You can then configure the alert policy to message this channel. Note: Only messaging one channel per alert policy is currently supported:
To disconnect Dagster Cloud from Slack, remove the Dagster Cloud app from your Slack workspace. Refer to Slack's documentation for more info and instructions. Once the app is removed, refresh the Alerts page in Dagster Cloud and the Connect to Slack option will be displayed.
Email - Email alerts can be sent to one or more recipients. For example:
In the alert policies YAML file, we'll define a policy that listens for jobs with a team tag of sales to succeed or fail. When this occurs, a notification will be sent to the sales-notification channel in the hooli workspace:
alert_policies:-name:"slack-alert-policy"description:"An alert policy to send a Slack notification to sales on job failure or success."tags:-key:"team"value:"sales"event_types:-"JOB_SUCCESS"-"JOB_FAILURE"notification_service:slack:slack_workspace_name:"hooli"slack_channel_name:"sales-notifications"
In the alert policies YAML file, we'll define a policy that listens for jobs with a level tag of critical to fail. When this occurs, an email notification will be sent to firstname.lastname@example.org and email@example.com:
alert_policies:-name:"email-alert-policy"description:"An alert policy to email company executives during job failure."tags:-key:"level"value:"critical"event_types:-"JOB_FAILURE"notification_service:email:email_addresses:-"firstname.lastname@example.org"-"email@example.com"
When creating an alert policy using the CLI, only certain event_types can be specified together. You can specify multiple job run-based event types together (JOB_SUCCESS, JOB_FAILURE), or a tick-based event type (TICK_FAILURE), but attempting to mix these will result in an error.