14 docs tagged with "compute"

The AWS EMR integration allows you to seamlessly integrate AWS EMR into your Dagster pipelines for petabyte-scale data processing using open source tools like Apache Spark, Hive, Presto, and more.

The AWS Glue integration enables you to initiate AWS Glue jobs directly from Dagster, seamlessly pass parameters to your code, and stream logs and structured messages back into Dagster.

Using the AWS Lambda integration with Dagster, you can leverage serverless functions to execute external code in your pipelines.

The Databricks integration enables you to initiate Databricks jobs directly from Dagster, seamlessly pass parameters to your code, and stream logs and structured messages back into Dagster.

Run runs external processes in docker containers directly from Dagster.

The community-supported `dagster-contrib-gcp` package provides integrations with Google Cloud Platform (GCP) services.

Integrate with GCP Dataproc.

The community-supported `dagster-nomad` package provides an integration with HashiCorp Nomad.

The community-supported `dagster-hex` package provides an integration with Hex.

Dagstermill eliminates the tedious "productionization" of Jupyter notebooks.

Launch Kubernetes pods and execute external code directly from Dagster.

The community-supported `dagster-modal` package provides an integration with Modal.

The `dagster-perian` integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment.

Configure and run Spark jobs.