84 docs tagged with "dagster-plus-feature"

Dagster+ is a managed orchestration platform for data engineering, offering Serverless and Hybrid deployment types with data cataloging, cost insights, authentication and RBAC, alerting, and branch deployment features.

Learn about Dagster+ alert policy types for assets, runs, code locations, automation, agent downtime, insights, and credit budget limits.

Define alerts and configure alert notification services to monitor critical events in your Dagster+ deployment.

The Dagster+ Amazon ECS agent manages container instances, enables communication with ECS service, and supports task lifecycle operations on AWS infrastructure.

The Dagster+ Hybrid architecture is the most flexible and secure way to deploy Dagster+, allowing you to run your user code in your environment while leveraging Dagster+'s infrastructure for orchestration and metadata management.

Use the Dagster+ asset catalog to view assets, access the global asset lineage, build dasbhoards, reload definitions, and search assets by asset key, compute kind, asset group, code location, and more.

With asset health criteria, you can quickly identify which datasets are performing well and which need attention in Dagster+.

Learn about RBAC, SSO, and SCIM provisioning with Dagster+.

With Branch Deployments, Dagster+ creates a corresponding branch deployment for each pull request to show what production will look like after the change is merged.

Dagster+ provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment.

Dagster+ Branch Deployments compare asset definitions in the branch deployment against the asset definitions in the base deployment, helping your team identify how changes in a pull request will impact data assets.

CI/CD branch deployments automate code integration and delivery for Dagster+ projects.

Reference page for the Dagster+ CI/CD configuration files branch_deployments.yml and deploy.yml.

Configure CI/CD for Dagster+ Hybrid using GitHub Actions, or non-GitHub providers with the dagster-cloud CLI.

Implement CI/CD for your Dagster+ Serverless deployment with GitHub, GitLab, or another Git provider.

Dagster+ configuration reference for Amazon ECS agents.

Configure Dagster+ alert notifications to trigger via email, Microsoft Teams, PagerDuty, or Slack.

Configure Microsoft Entra ID provisioning for Dagster+ to sync user information between Microsoft Entra ID and your Dagster+ deployment.

Configure SCIM provisioning in Dagster+ to sync user information between Okta and your Dagster+ deployment.

Create alert policies in Dagster+ via UI or dagster-cloud CLI on a per-deployment basis. Specify policy types, targets, and notification channels.

Flexibly build dashboards in the Dagster+ asset catalog scoped by tags, teams, owners, or asset groups in order to enable everyone on your team to focus on the assets that matter most to them.

Customize Dagster+ agent settings in dagster.yaml

The dagster_cloud.yaml file defines multiple code locations for Dagster+ projects.

Track and manage code location history and rollbacks in Dagster+.

Separate code locations allow you to deploy different Dagster projects that still roll up into a single Dagster+ deployment with one global lineage graph.

Dagster+ code must load from a single entry point and be able to be run in an environment where the dagster and dagster-cloud 0.13.2+ Python packages are installed, plus meet additional requirements for hybrid deployments.

In a Dagster+ Hybrid deployment, the orchestration control plane is run by Dagster+ while your Dagster code is executed within your environment.

The Dagster+ agent interacts with a specific set of IP addresses that you may need to allowlist in your infrastructure.

Dagster+ Serverless is a fully managed version of Dagster+ and is the easiest way to get started with Dagster.

Add these Dagster+ Serverless IP addresses to an allowlist for outbound requests to external services.

Deploy a Dagster+ agent on an Azure Kubernetes Service (AKS) cluster.

Deploy Dagster code to Azure Kubernetes Service using GitHub Actions and Azure Container Registry.

Manage Dagster+ deployment settings, environment variables, tokens, and more.

Learn to set up and configure the Dagster+ Docker agent.

Configure Docker agents in Dagster+.

Configure and run Dagster+ Docker agents to execute code within Docker containers.

Configure environment variables through the Dagster+ UI or with agent configuration to dynamically modify application behavior depending on environment.

Example Dagster+ YAML alert configuration.

Deploy a Dagster+ Amazon ECS agent in an existing VPC using CloudFormation.

Export Dagster+ Insights metrics using the Dagster GraphQL API.

Configure full deployment settings in Dagster+ using YAML.

Dagster+ full deployments are persistent, fully-featured deployments intended to perform actions on a recurring basis.

Get started with Dagster+ by creating a Dagster+ organization and choosing the Serverless or Hybrid deployment type.

Using real-time Dagster+ Insights, you can gain visibility into historical asset health, usage, and cost metrics, such as Dagster+ run duration and failures.

Create custom metrics from asset metadata to use Dagster+ Insights to perform historical aggregations on any data your assets can emit.

Automate deployment, scaling, and management of containerized Kubernetes applications with the Dagster+ agent.

Configure Dagster+ Kubernetes agents using Helm charts for per-deployment and per-location settings.

Set up the Dagster+ agent on a Kubernetes cluster using Helm. Configure secrets, manage deployments, and perform rolling upgrades.

Create and revoke agent tokens for authenticating Dagster+ hybrid agents.

Frequent use cases and troubleshooting tips for managing branch deployments when your organization has multiple Dagster+ deployments or environments.

Configure where Dagster+ compute logs are stored and manage masking of error messages in the Dagster+ UI.

Manage standalone Dagster+ full deployments with independent permissions.

Manage multiple projects with Dagster+ Hybrid deployments.

Manage team permissions in Dagster+ with role-based access control (RBAC).

Viewing, creating, editing, and revoking user tokens in Dagster+.

Dagster+ allows you to grant specific permissions to your organization's users with role-based access control (RBAC), ensuring that Dagster users have access only to what they need.

Manually set up and deploy a Dagster+ Amazon ECS agent.

Deploy Dagster+ on Azure using AKS, ACR, Azure Blob Storage, and Azure Key Vault.

Set up and deploy a Dagster+ Amazon ECS agent in a new VPC using CloudFormation with Dagster+.

Configure OneLogin to use single sign-on (SSO) with your Dagster+ organization.

Dagster+ imposes rate limits of 40,000 user log events per minute and 100MB of events per minute, with automatic retries for requests that exceed limits.

Retrieve secrets and credentials from Azure Key Vault in an Azure Kubernetes Service (AKS) cluster.

Manage user and team permissions in Dagster+ with role-based access control (RBAC).

Configure and run a local Dagster+ agent for testing before scaling with your preferred cloud service provider.

Configure multiple Dagster+ agents for redundancy or isolation in the same environment or across different environments using Docker, Kubernetes, or Amazon ECS.

Automatically sync user information from your identity provider to Dagster+ and back with SCIM provisioning.

Dagster+ Serverless run isolation offers isolated runs for production with dedicated resources, and non-isolated runs in a standing, shared container for faster development.

Customizing the Dagster+ Serverless runtime environment.

Dagster+ Serverless secures data and secrets with container sandboxing and per-customer registries. Adjust I/O managers for PII, PHI, or GDPR compliance.

Configure environment variables in Dagster+ Hybrid deployments using the hybrid agent's configuration.

Configure environment variables in the Dagster+ UI with secure storage.

Configure branch deployments for a code location in Dagster+ using GitHub, GitLab, or the dagster-cloud CLI.

Configure Google Workspace to use single sign-on (SSO) with your Dagster+ organization.

Configure Microsoft Entra ID (formerly Azure Active Directory) to use SSO with your Dagster+ organization.

Configure Okta to use single sign-on (SSO) with your Dagster+ organization.

Configure PingOne to use single sign-on (SSO) with your Dagster+ organization.

Configure single sign-on (SSO) for your Dagster+ organization.

Store Dagster+ compute logs in Azure Blob Storage or Azure Data Lake Storage.

Test your code in Dagster+ using branch deployments without impacting production data.

Managing user and agent tokens in Dagster+.

Upgrade your Amazon ECS CloudFormation template to use the newest image and Dagster version.

Role-based access control (RBAC) enables you to grant specific permissions to users in your Dagster+ organization, ensuring that users only have access to what they need.

Setting up branch deployments with a general continuous integration (CI) or git platform, using the dagster-cloud CLI.

The Dagster+ audit log enables Dagster+ Pro organization admins to track and attribute changes to their Dagster deployment with the UI or Dagster+ GraphQL API.