Welcome to Dagster! If you’re here, you probably have questions about data, orchestration, Dagster, and how everything fits together.
Let’s start with the basics: What’s an orchestrator?
Orchestrators allow you to automatically execute a series of steps and track the results. They not only enable you and your team to see how long steps take to run or how they connect to other steps, but to understand when, where, and why things break.
What if you’re working with data? Unlike generic orchestrators that may need to accommodate a variety of use cases, data orchestrators focus on building and managing data pipelines.
Data orchestrators allow you to automatically run, in the correct order, the steps in data pipelines that move, process, and transform data. A data orchestrator helps to ensure that your data assets - such as database tables, reports, language learning models, and so on - are accurate and up-to-date.
That being said, you might be asking yourself...
Dagster models pipelines in terms of the data assets they produce and consume, which, by default, brings order and observability to your data platform. Assets in Dagster can model data produced by any system, such as dbt models, Snowflake tables, or even CSV files.
With Dagster, you declare functions in Python that you want to run and the data assets that those functions produce or update. For example, the following image is an example of a graph with several assets:
Dagster’s asset-centric approach to building data pipelines makes it easy to:
Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.
Additionally, Dagster is accompanied by a sleek, modern, web-based UI. This ‘single pane of glass’ serves both engineers and the stakeholders they support: engineers can use the UI to inspect the data pipelines they create in code, and stakeholders can observe and monitor their data.
If you want to try running Dagster yourself, check out the Hello, Dagster! quickstart.
Ready to dive in? Depending on your learning preferences, there are a few ways to get started: