Spark (dagster-spark)¶

class dagster_spark.SparkOpError[source]¶

dagster_spark.define_spark_config()[source]¶

Spark configuration.

See the Spark documentation for reference:

https://spark.apache.org/docs/latest/submitting-applications.html

dagster_spark.create_spark_op(name, main_class, description=None, required_resource_keys=frozenset({'spark'}))[source]¶

dagster_spark.construct_spark_shell_command(application_jar, main_class, master_url=None, spark_conf=None, deploy_mode=None, application_arguments=None, spark_home=None)[source]¶

Constructs the spark-submit command for a Spark job.