Dagster & Census (Component)
This is a community-maintained integration. To report bugs or leave feedback, open an issue in the Dagster community integrations repo.
The dagster-census library provides a CensusComponent which can be used to easily represent Census syncs as assets in Dagster.
CensusComponent is a state-backed component, which fetches and caches Census workspace metadata. For information on managing component state, see Configuring state-backed components.
1. Prepare a Dagster project
To begin, you'll need a Dagster project. You can use an existing components-ready project or create a new one:
uvx create-dagster project my-project && cd my-project/src
Activate the project virtual environment:
source ../.venv/bin/activate
Finally, add the dagster-census library to the project:
uv add dagster-census
2. Scaffold a Census component definition
Now that you have a Dagster project, you can scaffold a Census component definition. You'll need to provide your Census API key, which you can set as an environment variable on the command line:
dg scaffold defs dagster_census.CensusComponent census_ingest
The dg scaffold defs call will generate a defs.yaml file:
.
└── census_ingest
└── defs.yaml
YAML configuration
In its scaffolded form, the defs.yaml file contains the configuration for your Census workspace:
type: dagster_census.CensusComponent
attributes:
workspace:
api_key: '{{ env.CENSUS_API_KEY }}'
3. Check the component configuration
You can check the configuration of your component with dg list defs:
$ dg list defs
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ census_sync_1 │ default │ │ census │ Asset generated from Census sync 12345 │ │
│ │ ├───────────────────┼─────────┼──────┼─────────┼──────────────────────────────────────────┤ │
│ │ │ census_sync_2 │ default │ │ census │ Asset generated from Census sync 67890 │ │
│ │ └───────────────────┴─────────┴──────┴─────────┴──────────────────────────────────────────┘ │
└──────── ───┴─────────────────────────────────────────────────────────────────────────────────────────────┘
4. Select specific syncs
You can select specific Census syncs to include in your component using the sync_selector key. This allows you to filter which syncs are represented as assets:
type: dagster_census.CensusComponent
attributes:
workspace:
api_key: "{{ env.CENSUS_API_KEY }}"
sync_selector:
by_name:
- census_sync_1
$ dg list defs
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ census_sync_1 │ default │ │ census │ Asset generated from Census sync 12345 │ │
│ │ └───────────────────┴─────────┴──────┴─────────┴──────────────────────────────────────────┘ │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘
You can also select syncs by ID:
type: dagster_census.CensusComponent
attributes:
workspace:
api_key: "{{ env.CENSUS_API_KEY }}"
sync_selector:
by_id:
- 12345
- 67890
About Census
Census syncs data from your cloud warehouse to the SaaS tools your organization uses. It allows everyone in your organization to take action with good data, no custom scripts or API integrations required.