Ask AI

Spark (dagster-spark)

class dagster_spark.SparkOpError[source]
dagster_spark.define_spark_config()[source]

Spark configuration.

See the Spark documentation for reference:

https://spark.apache.org/docs/latest/submitting-applications.html

dagster_spark.create_spark_op(name, main_class, description=None, required_resource_keys=frozenset({'spark'}))[source]
dagster_spark.construct_spark_shell_command(application_jar, main_class, master_url=None, spark_conf=None, deploy_mode=None, application_arguments=None, spark_home=None)[source]

Constructs the spark-submit command for a Spark job.

Legacy

dagster_spark.spark_resource ResourceDefinition[source]