GCP (dagster_gcp)

BigQuery

class dagster_gcp.BigQueryError[source]
dagster_gcp.bigquery_resource ResourceDefinition[source]
dagster_gcp.bq_create_dataset(*args, **kwargs)[source]

BigQuery Create Dataset.

This solid encapsulates creating a BigQuery dataset.

Expects a BQ client to be provisioned in resources as context.resources.bigquery.

dagster_gcp.bq_delete_dataset(*args, **kwargs)[source]

BigQuery Delete Dataset.

This solid encapsulates deleting a BigQuery dataset.

Expects a BQ client to be provisioned in resources as context.resources.bigquery.

dagster_gcp.bq_solid_for_queries(sql_queries)[source]

Executes BigQuery SQL queries.

Expects a BQ client to be provisioned in resources as context.resources.bigquery.

dagster_gcp.import_df_to_bq(*args, **kwargs)[source]
dagster_gcp.import_file_to_bq(*args, **kwargs)[source]
dagster_gcp.import_gcs_paths_to_bq(*args, **kwargs)[source]

Dataproc

dagster_gcp.dataproc_solid(*args, **kwargs)[source]
dagster_gcp.dataproc_resource ResourceDefinition[source]

GCS

dagster_gcp.gcs_system_storage SystemStorageDefinition[source]
dagster_gcp.gcs.gcs_intermediate_storage IntermediateStorageDefinition[source]
dagster_gcp.gcs_resource ResourceDefinition[source]