Dagster & Looker (Component)
This feature is considered in a beta stage. It is still being tested and may change. For more information, see the API lifecycle stages documentation.
The dagster-looker library provides a LookerComponent which can be used to easily represent Looker dashboards and explores as assets in Dagster.
LookerComponent is a state-backed component, which fetches and caches Looker instance metadata. For information on managing component state, see Configuring state-backed components.
1. Prepare a Dagster project
To begin, you'll need a Dagster project. You can use an existing components-ready project or create a new one:
uvx create-dagster project my-project && cd my-project/src
Activate the project virtual environment:
source ../.venv/bin/activate
Finally, add the dagster-looker library to the project:
uv add dagster-looker
2. Scaffold a Looker component definition
Now that you have a Dagster project, you can scaffold a Looker component definition:
dg scaffold defs dagster_looker.LookerComponent looker_ingest
Creating defs at /.../my-project/src/my_project/defs/looker_ingest.
The dg scaffold defs call will generate a defs.yaml file:
tree my_project/defs
my_project/defs
├── __init__.py
└── looker_ingest
└── defs.yaml
2 directories, 2 files
3. Configure your Looker instance
Update the defs.yaml file with your Looker instance connection details. You'll need to provide your base URL, client ID, and client secret. For more information on creating API credentials, see the Looker API documentation.
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: "{{ env.LOOKER_BASE_URL }}"
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ my_dashboard_1 │ default │ my_model::my_explore │ dashboard │ │ │
│ │ │ │ │ │ looker │ │ │
│ │ ├──────────────────────┼─────────┼──────────────────────┼───────────┼─────────────┤ │
│ │ │ my_model::my_explore │ default │ view/my_view │ explore │ │ │
│ │ │ │ │ │ looker │ │ │
│ │ ├──────────────────────┼─────────┼──────────────────────┼───────────┼─────────────┤ │
│ │ │ view/my_view │ default │ │ │ │ │
│ │ └──────────────────────┴─────────┴──────────────────────┴───────────┴─────────────┘ │
└─────────┴───────────────────────────────────────────────────────────────────── ────────────────┘
4. Filter Looker content
You can filter which Looker dashboards and explores are loaded using the looker_filter key. For example, you can load only dashboards from specific folders and only explores used in those dashboards:
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: "{{ env.LOOKER_BASE_URL }}"
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
looker_filter:
dashboard_folders:
- ["Shared"]
only_fetch_explores_used_in_dashboards: true
5. Customize Looker asset metadata
You can customize the metadata and grouping of Looker assets using the translation key:
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: "{{ env.LOOKER_BASE_URL }}"
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
translation:
group_name: looker_data
description: "Looker {{ data.structure_type.value }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │
│ │ │ my_dashboard_1 │ looker_data │ my_model::my_explore │ dashboard │ Looker dashboard │ │
│ │ │ │ │ │ looker │ │ │
│ │ ├──────────────────────┼─────────────┼──────────────────────┼───────────┼──────────────────┤ │
│ │ │ my_model::my_explore │ looker_data │ view/my_view │ explore │ Looker explore │ │
│ │ │ │ │ │ looker │ │ │
│ │ ├──────────────────────┼─────────────┼──────────────────────┼───────────┼──────────────────┤ │
│ │ │ view/my_view │ default │ │ │ │ │
│ │ └──────────────────────┴─────────────┴──────────────────────┴───────────┴──────────────────┘ │
└─────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
Customize specific data types
You may also specify distinct translation behavior for specific data types. For example, you can add a tag to all dashboards:
type: dagster_looker.LookerComponent
attributes:
looker_resource:
base_url: "{{ env.LOOKER_BASE_URL }}"
client_id: "{{ env.LOOKER_CLIENT_ID }}"
client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
translation:
for_dashboard:
tags:
is_dashboard: "true"
dg list defs --columns name,kinds,tags
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Kinds ┃ Tags ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━ ━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ my_dashboard_1 │ dashboard │ "is_dashboard"="true" │ │
│ │ │ │ looker │ │ │
│ │ ├──────────────────────┼───────────┼───────────────────────┤ │
│ │ │ my_model::my_explore │ explore │ │ │
│ │ │ │ looker │ │ │
│ │ ├──────────────────────┼───────────┼───────────────────────┤ │
│ │ │ view/my_view │ │ │ │
│ │ └──────────────────────┴───────────┴───────────────────────┘ │
└─────────┴──────────────────────────────────────────────────────────────┘