Dagster & Qdrant

The dagster-qdrant library lets you integrate Qdrant's vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster.

Installation

pip install dagster dagster-qdrant

Example

from dagster_qdrant import QdrantConfig, QdrantResource

import dagster as dg


@dg.asset
def my_table(qdrant_resource: QdrantResource):
    with qdrant_resource.get_client() as qdrant:
        qdrant.add(
            collection_name="test_collection",
            documents=[
                "This is a document about oranges",
                "This is a document about pineapples",
                "This is a document about strawberries",
                "This is a document about cucumbers",
            ],
        )
        results = qdrant.query(
            collection_name="test_collection", query_text="hawaii", limit=3
        )


defs = dg.Definitions(
    assets=[my_table],
    resources={
        "qdrant_resource": QdrantResource(
            config=QdrantConfig(
                host="xyz-example.eu-central.aws.cloud.qdrant.io",
                api_key="<your-api-key>",
            )
        )
    },
)

About Qdrant

Qdrant (read: quadrant) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage vectors with additional payload and extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications.

Learn more from the Qdrant documentation.