Skip to main content

dagster-clickhouse library

This library provides an integration with ClickHouse using the native protocol via clickhouse-driver.

Related guides:

Resources​

dagster_clickhouse.ClickhouseResource ResourceDefinition

Resource for interacting with a ClickHouse database via clickhouse-driver (native protocol).

Examples:

from dagster import Definitions, asset
from dagster_clickhouse import ClickhouseResource

@asset
def my_table(clickhouse: ClickhouseResource):
    with clickhouse.get_connection() as client:
        client.execute("SELECT * FROM my_database.my_table LIMIT 1")

Definitions(
    assets=[my_table],
    resources={"clickhouse": ClickhouseResource(host="localhost")},
)

I/O manager​

dagster_clickhouse.ClickhouseIOManager IOManagerDefinition

Base class for an IO manager that reads inputs from and writes outputs to ClickHouse.

Asset and op metadata schema (and the I/O manager schema config) refer to the ClickHouse database that contains the table.

The database field is the default database on the ClickHouse connection (server-side), not the database that qualifies table names in Dagster metadata (that is schema).

Examples:

Subclass and implement type_handlers() with handlers from dagster-clickhouse-pandas or dagster-clickhouse-polars.

dagster_clickhouse.build_clickhouse_io_manager

Builds an IO manager definition that reads inputs from and writes outputs to ClickHouse.

TableSlice.schema is mapped to a ClickHouse database name (not a separate schema layer).

Parameters:

  • type_handlers – Handlers defining how to translate between ClickHouse tables and in-memory types (e.g. a Pandas DataFrame). If only one handler is provided, it becomes the default load type when no annotation is present.
  • default_load_type – Type to use when an input has no type annotation.

Returns: IOManagerDefinition

Database utilities​

class dagster_clickhouse.ClickhouseDbClient
dagster_clickhouse.format_clickhouse_table_fqn

Return the fully quoted database.table name (Dagster schema = ClickHouse database).

Component​

class dagster_clickhouse.ClickhouseQueryComponent
preview

This API is currently in preview, and may have breaking changes in patch version releases. This API is not considered ready for production use.

A ClickHouse connection component for TemplatedSqlComponent and other SQL tooling.