Spark (dagster-spark)

class dagster_spark.SparkOpError [source]

dagster_spark.define_spark_config [source]

Spark configuration.

See the Spark documentation for reference: https://spark.apache.org/docs/latest/submitting-applications.html

dagster_spark.create_spark_op [source]

dagster_spark.construct_spark_shell_command [source]

Constructs the spark-submit command for a Spark job.

Legacy

dagster_spark.spark_resource ResourceDefinition [source]